Auto-refresh KB pricing, align skill with Anthropic best practices

Why this set of changes:
- KB pricing was drifting silently — domain files (database.yaml,
  storage.yaml, etc.) had prices 30-800% off the live Oracle API and
  nobody read them. The skill was auditing as stale on every check
  with no path to fix it.
- The skill itself violated Anthropic's spec (`name` field had
  uppercase/spaces) and was over the 500-line guideline (647 lines),
  hurting discovery and load performance.
- Welcome flow occasionally improvised the menu instead of reading
  SKILL.md, missing options.

Pricing — single source of truth, fully automated:
- Extend tools/refresh_sku_catalog.py with --refresh-domain compute,
  pulls shape-level prices from the Oracle public pricing API
  (apexapps.oracle.com), preserves manual fields (notes, GPU specs,
  free-tier annotations, estimation_helpers), recomputes derived
  monthly values, and protects $0 free-tier prices from overwrite.
- Delete 12 redundant pricing/<domain>.yaml files. They duplicated
  oci-sku-catalog.yaml with worse abstractions and were nobody's
  source of truth (no tool consumed them).
- Migrate the genuinely valuable knowledge from those 12 files
  (billing models, BYOL rules, free-tier rules, ECPU vs OCPU,
  X11M elastic model, hyperscaler comparisons, service nuances)
  into kb/field-knowledge/pricing-knowledge.yaml — non-numeric,
  no refresh needed.
- Result: pricing freshness check goes from 13 stale files to 0.

KB freshness automation:
- Add tools/kb_freshness.py — wrapper around kb_linter.check_freshness()
  with --check, --auto-refresh, --json, --quiet modes. Bridges stale
  files to their refresh tools (SKU catalog, compute domain, arch
  center). Wired into the welcome flow as a pre-flight banner that
  asks the user before refreshing.
- Fix pre-existing kb_linter bug: it crashed on the 45 multi-doc
  YAML files (frontmatter + body pattern) because it used safe_load
  instead of safe_load_all. Freshness check was effectively dead.
- Standardize timestamp field: linter now accepts last_verified,
  last_updated, and last_refreshed; refresh_arch_catalog writes
  last_verified instead of last_refreshed.
- Add make freshness / make freshness-refresh targets.

Skill alignment with Anthropic Agent Skills best practices:
- Rename `name: OCI Deal Accelerator` to `oci-deal-accelerator`
  to comply with the [a-z0-9-]{1,64} spec.
- Refactor SKILL.md from 674 to 445 lines via progressive disclosure:
  extract WA review output format, ECAL readiness format, and output
  conventions into docs/skill/*.md referenced from the main file.
- Add scripts/sync-skill.py + make sync-skill: source of truth is
  root SKILL.md, .agents/skills/oci-deal-accelerator/SKILL.md is
  auto-generated. make lint validates sync.
- Add evaluations/ with 3 manual baseline scenarios (welcome-flow,
  full-proposal, wa-review) per the Anthropic best-practices guidance
  to "build evaluations first."

Welcome flow hardening:
- Tighten CLAUDE.md to MANDATE reading SKILL.md before showing the
  menu (no improvising), and document the freshness pre-flight check
  with the ask-before-refresh user flow.
- Update SKILL.md welcome flow to instruct: parse kb_freshness JSON,
  show banner with stale count + oldest file, prompt user to refresh
  (only when an automated tool exists), fall back silently on errors.

Linter hygiene (zero remaining issues):
- Expand config/kb-tags.yaml taxonomy with features, operations,
  metrics, limitations sections covering 31 previously-unknown tags
  used in field findings (rac, ecpu, refreshable-clone, hnsw, etc.).
- Assign owners for kb/compatibility/, kb/competitive/,
  kb/well-architected/ (Diego Cabrera as default until team grows);
  kb/pricing/ marked as "Auto-refreshed" since it no longer needs
  human ownership.
- kb_linter accepts top-level `date` as fallback for contributor
  block; migrate FF-202603-008 from legacy `reported_by` to
  contributor block.
- Result: linter goes from 45 issues to 0.

Other:
- Recompute estimation_helpers monthly values in compute.yaml after
  the price refresh (they were derived from the old E5/A1 numbers).
- Add kb/README.md — contributor guide (directory map, frontmatter
  spec, refresh tooling, review cadence).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-08 23:59:32 -03:00
parent 69b0ccb4b8
commit ca93a0aa4e
37 changed files with 1689 additions and 1639 deletions

26
evaluations/README.md Normal file
View File

@@ -0,0 +1,26 @@
# Evaluations
Baseline evaluation scenarios for the OCI Deal Accelerator skill, following
the format recommended in the [Anthropic Agent Skills best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices#build-evaluations-first).
These are **manual evaluations** — there is no automated test runner yet.
Use them as a regression checklist after non-trivial changes to `SKILL.md`,
`CLAUDE.md`, or the welcome flow.
## How to run
1. Open a fresh Claude Code conversation in this repo.
2. For each `*.json` file in this directory:
- Read the `query` field — paste it as the user message (or simulate the trigger described in the file).
- If `files` is non-empty, attach those files as context.
- Observe the assistant's behavior.
- Manually verify each item in `expected_behavior` is met.
3. Note any failures — file an issue or update the skill accordingly.
## Scenarios
| File | Purpose |
|---|---|
| `welcome-flow.json` | Greeting → exact 14-option menu (verbatim from SKILL.md) |
| `full-proposal.json` | Discovery notes → bypass menu, run full proposal flow |
| `wa-review.json` | WA review request → scorecard format + saved YAML files |

View File

@@ -0,0 +1,18 @@
{
"skills": ["oci-deal-accelerator"],
"query": "Customer: ACME Bank, retail banking, 200 branches across LATAM. Driver: migrating core banking from on-prem Exadata to OCI for cost reduction and DR. Current: 4-node RAC on X8M, ~12 TB DB, 2000 TPS peak, RTO 4h / RPO 15min. Compliance: PCI-DSS, local data residency. Timeline: 6 months. Budget: tight, exploring BYOL. They are also evaluating AWS RDS Custom for Oracle.",
"files": [],
"expected_behavior": [
"Detects this is discovery notes and skips the welcome menu (no 14-option list shown)",
"Goes directly into the full proposal flow (option 1 behavior)",
"Builds a workload profile reflecting ACME Bank's stated requirements (does NOT invent services they did not mention)",
"Recommends an architecture that addresses RTO/RPO, PCI-DSS, data residency, and BYOL — using kb/services/, kb/patterns/, and kb/well-architected/",
"Cites at least one Oracle Architecture Center reference architecture from kb/architecture-center/catalog.yaml when there is a strong match",
"Generates a slide deck (.pptx) as the primary deliverable, saved to examples/output-acme-bank-*/",
"Includes a Cost Estimate slide that distinguishes BYOL vs PAYG using kb/pricing/",
"Mentions the AWS RDS Custom for Oracle competitive context honestly using kb/competitive/aws-mapping.yaml",
"Does NOT add observability/security services that ACME did not request, beyond what is required for stated compliance",
"Asks clarifying questions when input is genuinely ambiguous, instead of inventing answers",
"Surfaces relevant gotchas from kb/field-knowledge/ (e.g., Exadata-to-cloud migration risks)"
]
}

View File

@@ -0,0 +1,20 @@
{
"skills": ["oci-deal-accelerator"],
"query": "5",
"follow_up": "Single-region OCI architecture: ADB-S 8 ECPU in São Paulo, 2 VM.Standard.E5.Flex web tier behind a public load balancer, Object Storage for media. No backups configured, no FastConnect. Customer: SmallShop, e-commerce, no compliance requirements stated.",
"files": [],
"expected_behavior": [
"After the user picks 5, asks for the architecture description (does not skip to scoring)",
"Reads docs/skill/wa-review-format.md before producing the scorecard",
"Parses the input into a workload profile YAML and architecture YAML",
"Runs scripts/validate-architecture.py with both files",
"Produces the formatted terminal scorecard with the exact banner format from docs/skill/wa-review-format.md (5 pillars, X/Y passed, HIGH/MEDIUM/LOW counts)",
"Saves three YAML files under examples/: <customer>-wa-scorecard.yaml, <customer>-wa-architecture.yaml, <customer>-wa-workload-profile.yaml",
"Lists the saved files at the end of the review",
"Identifies the obvious gaps (no backups, no DR, no monitoring, single-AD) as HIGH severity",
"Presents HIGH gaps as a markdown table grouped by pillar",
"Presents MEDIUM gaps as compact bullets, not a table",
"Offers the after-WA-review menu with options [A] [B] [C] [D]",
"If the user picks [A], remediates the EXISTING architecture (adds encryption, backups, etc. to the actual services SmallShop has) — does NOT replace it with a generic 'ideal' architecture, does NOT invent services SmallShop did not mention"
]
}

View File

@@ -0,0 +1,16 @@
{
"skills": ["oci-deal-accelerator"],
"query": "hola",
"files": [],
"expected_behavior": [
"Reads SKILL.md before showing anything to the user (does not improvise the menu from memory or folder structure)",
"Runs python tools/kb_freshness.py --check --json as a pre-flight step",
"If freshness reports stale files with refreshable=true, prepends a banner asking the user [y/N] before proceeding",
"If freshness reports stale files but none are refreshable, prepends a non-blocking informational banner without asking",
"Shows the welcome banner exactly as defined in SKILL.md (no paraphrasing)",
"Shows all 14 capability menu options in the correct 6 sections: DESIGN & PROPOSE (1-4), VALIDATE & CHECK (5-7), STRATEGY & BUSINESS (8), KNOWLEDGE BASE (9-11), ECAL GOVERNANCE (12), SA TOOLS (13-14)",
"Does NOT invent options that are not in SKILL.md",
"Does NOT omit options that are in SKILL.md",
"If the freshness tool errors out, falls back to showing the welcome message silently — never blocks the user on tooling failures"
]
}