forked from diegoecab/oci-deal-accelerator
Auto-refresh KB pricing, align skill with Anthropic best practices
Why this set of changes:
- KB pricing was drifting silently — domain files (database.yaml,
storage.yaml, etc.) had prices 30-800% off the live Oracle API and
nobody read them. The skill was auditing as stale on every check
with no path to fix it.
- The skill itself violated Anthropic's spec (`name` field had
uppercase/spaces) and was over the 500-line guideline (647 lines),
hurting discovery and load performance.
- Welcome flow occasionally improvised the menu instead of reading
SKILL.md, missing options.
Pricing — single source of truth, fully automated:
- Extend tools/refresh_sku_catalog.py with --refresh-domain compute,
pulls shape-level prices from the Oracle public pricing API
(apexapps.oracle.com), preserves manual fields (notes, GPU specs,
free-tier annotations, estimation_helpers), recomputes derived
monthly values, and protects $0 free-tier prices from overwrite.
- Delete 12 redundant pricing/<domain>.yaml files. They duplicated
oci-sku-catalog.yaml with worse abstractions and were nobody's
source of truth (no tool consumed them).
- Migrate the genuinely valuable knowledge from those 12 files
(billing models, BYOL rules, free-tier rules, ECPU vs OCPU,
X11M elastic model, hyperscaler comparisons, service nuances)
into kb/field-knowledge/pricing-knowledge.yaml — non-numeric,
no refresh needed.
- Result: pricing freshness check goes from 13 stale files to 0.
KB freshness automation:
- Add tools/kb_freshness.py — wrapper around kb_linter.check_freshness()
with --check, --auto-refresh, --json, --quiet modes. Bridges stale
files to their refresh tools (SKU catalog, compute domain, arch
center). Wired into the welcome flow as a pre-flight banner that
asks the user before refreshing.
- Fix pre-existing kb_linter bug: it crashed on the 45 multi-doc
YAML files (frontmatter + body pattern) because it used safe_load
instead of safe_load_all. Freshness check was effectively dead.
- Standardize timestamp field: linter now accepts last_verified,
last_updated, and last_refreshed; refresh_arch_catalog writes
last_verified instead of last_refreshed.
- Add make freshness / make freshness-refresh targets.
Skill alignment with Anthropic Agent Skills best practices:
- Rename `name: OCI Deal Accelerator` to `oci-deal-accelerator`
to comply with the [a-z0-9-]{1,64} spec.
- Refactor SKILL.md from 674 to 445 lines via progressive disclosure:
extract WA review output format, ECAL readiness format, and output
conventions into docs/skill/*.md referenced from the main file.
- Add scripts/sync-skill.py + make sync-skill: source of truth is
root SKILL.md, .agents/skills/oci-deal-accelerator/SKILL.md is
auto-generated. make lint validates sync.
- Add evaluations/ with 3 manual baseline scenarios (welcome-flow,
full-proposal, wa-review) per the Anthropic best-practices guidance
to "build evaluations first."
Welcome flow hardening:
- Tighten CLAUDE.md to MANDATE reading SKILL.md before showing the
menu (no improvising), and document the freshness pre-flight check
with the ask-before-refresh user flow.
- Update SKILL.md welcome flow to instruct: parse kb_freshness JSON,
show banner with stale count + oldest file, prompt user to refresh
(only when an automated tool exists), fall back silently on errors.
Linter hygiene (zero remaining issues):
- Expand config/kb-tags.yaml taxonomy with features, operations,
metrics, limitations sections covering 31 previously-unknown tags
used in field findings (rac, ecpu, refreshable-clone, hnsw, etc.).
- Assign owners for kb/compatibility/, kb/competitive/,
kb/well-architected/ (Diego Cabrera as default until team grows);
kb/pricing/ marked as "Auto-refreshed" since it no longer needs
human ownership.
- kb_linter accepts top-level `date` as fallback for contributor
block; migrate FF-202603-008 from legacy `reported_by` to
contributor block.
- Result: linter goes from 45 issues to 0.
Other:
- Recompute estimation_helpers monthly values in compute.yaml after
the price refresh (they were derived from the old E5/A1 numbers).
- Add kb/README.md — contributor guide (directory map, frontmatter
spec, refresh tooling, review cadence).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
26
evaluations/README.md
Normal file
26
evaluations/README.md
Normal file
@@ -0,0 +1,26 @@
|
||||
# Evaluations
|
||||
|
||||
Baseline evaluation scenarios for the OCI Deal Accelerator skill, following
|
||||
the format recommended in the [Anthropic Agent Skills best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices#build-evaluations-first).
|
||||
|
||||
These are **manual evaluations** — there is no automated test runner yet.
|
||||
Use them as a regression checklist after non-trivial changes to `SKILL.md`,
|
||||
`CLAUDE.md`, or the welcome flow.
|
||||
|
||||
## How to run
|
||||
|
||||
1. Open a fresh Claude Code conversation in this repo.
|
||||
2. For each `*.json` file in this directory:
|
||||
- Read the `query` field — paste it as the user message (or simulate the trigger described in the file).
|
||||
- If `files` is non-empty, attach those files as context.
|
||||
- Observe the assistant's behavior.
|
||||
- Manually verify each item in `expected_behavior` is met.
|
||||
3. Note any failures — file an issue or update the skill accordingly.
|
||||
|
||||
## Scenarios
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `welcome-flow.json` | Greeting → exact 14-option menu (verbatim from SKILL.md) |
|
||||
| `full-proposal.json` | Discovery notes → bypass menu, run full proposal flow |
|
||||
| `wa-review.json` | WA review request → scorecard format + saved YAML files |
|
||||
18
evaluations/full-proposal.json
Normal file
18
evaluations/full-proposal.json
Normal file
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"skills": ["oci-deal-accelerator"],
|
||||
"query": "Customer: ACME Bank, retail banking, 200 branches across LATAM. Driver: migrating core banking from on-prem Exadata to OCI for cost reduction and DR. Current: 4-node RAC on X8M, ~12 TB DB, 2000 TPS peak, RTO 4h / RPO 15min. Compliance: PCI-DSS, local data residency. Timeline: 6 months. Budget: tight, exploring BYOL. They are also evaluating AWS RDS Custom for Oracle.",
|
||||
"files": [],
|
||||
"expected_behavior": [
|
||||
"Detects this is discovery notes and skips the welcome menu (no 14-option list shown)",
|
||||
"Goes directly into the full proposal flow (option 1 behavior)",
|
||||
"Builds a workload profile reflecting ACME Bank's stated requirements (does NOT invent services they did not mention)",
|
||||
"Recommends an architecture that addresses RTO/RPO, PCI-DSS, data residency, and BYOL — using kb/services/, kb/patterns/, and kb/well-architected/",
|
||||
"Cites at least one Oracle Architecture Center reference architecture from kb/architecture-center/catalog.yaml when there is a strong match",
|
||||
"Generates a slide deck (.pptx) as the primary deliverable, saved to examples/output-acme-bank-*/",
|
||||
"Includes a Cost Estimate slide that distinguishes BYOL vs PAYG using kb/pricing/",
|
||||
"Mentions the AWS RDS Custom for Oracle competitive context honestly using kb/competitive/aws-mapping.yaml",
|
||||
"Does NOT add observability/security services that ACME did not request, beyond what is required for stated compliance",
|
||||
"Asks clarifying questions when input is genuinely ambiguous, instead of inventing answers",
|
||||
"Surfaces relevant gotchas from kb/field-knowledge/ (e.g., Exadata-to-cloud migration risks)"
|
||||
]
|
||||
}
|
||||
20
evaluations/wa-review.json
Normal file
20
evaluations/wa-review.json
Normal file
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"skills": ["oci-deal-accelerator"],
|
||||
"query": "5",
|
||||
"follow_up": "Single-region OCI architecture: ADB-S 8 ECPU in São Paulo, 2 VM.Standard.E5.Flex web tier behind a public load balancer, Object Storage for media. No backups configured, no FastConnect. Customer: SmallShop, e-commerce, no compliance requirements stated.",
|
||||
"files": [],
|
||||
"expected_behavior": [
|
||||
"After the user picks 5, asks for the architecture description (does not skip to scoring)",
|
||||
"Reads docs/skill/wa-review-format.md before producing the scorecard",
|
||||
"Parses the input into a workload profile YAML and architecture YAML",
|
||||
"Runs scripts/validate-architecture.py with both files",
|
||||
"Produces the formatted terminal scorecard with the exact banner format from docs/skill/wa-review-format.md (5 pillars, X/Y passed, HIGH/MEDIUM/LOW counts)",
|
||||
"Saves three YAML files under examples/: <customer>-wa-scorecard.yaml, <customer>-wa-architecture.yaml, <customer>-wa-workload-profile.yaml",
|
||||
"Lists the saved files at the end of the review",
|
||||
"Identifies the obvious gaps (no backups, no DR, no monitoring, single-AD) as HIGH severity",
|
||||
"Presents HIGH gaps as a markdown table grouped by pillar",
|
||||
"Presents MEDIUM gaps as compact bullets, not a table",
|
||||
"Offers the after-WA-review menu with options [A] [B] [C] [D]",
|
||||
"If the user picks [A], remediates the EXISTING architecture (adds encryption, backups, etc. to the actual services SmallShop has) — does NOT replace it with a generic 'ideal' architecture, does NOT invent services SmallShop did not mention"
|
||||
]
|
||||
}
|
||||
16
evaluations/welcome-flow.json
Normal file
16
evaluations/welcome-flow.json
Normal file
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"skills": ["oci-deal-accelerator"],
|
||||
"query": "hola",
|
||||
"files": [],
|
||||
"expected_behavior": [
|
||||
"Reads SKILL.md before showing anything to the user (does not improvise the menu from memory or folder structure)",
|
||||
"Runs python tools/kb_freshness.py --check --json as a pre-flight step",
|
||||
"If freshness reports stale files with refreshable=true, prepends a banner asking the user [y/N] before proceeding",
|
||||
"If freshness reports stale files but none are refreshable, prepends a non-blocking informational banner without asking",
|
||||
"Shows the welcome banner exactly as defined in SKILL.md (no paraphrasing)",
|
||||
"Shows all 14 capability menu options in the correct 6 sections: DESIGN & PROPOSE (1-4), VALIDATE & CHECK (5-7), STRATEGY & BUSINESS (8), KNOWLEDGE BASE (9-11), ECAL GOVERNANCE (12), SA TOOLS (13-14)",
|
||||
"Does NOT invent options that are not in SKILL.md",
|
||||
"Does NOT omit options that are in SKILL.md",
|
||||
"If the freshness tool errors out, falls back to showing the welcome message silently — never blocks the user on tooling failures"
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user