Diagram tooling: make targets, CI gate, LLM-rewrite hook, refresh docs

Implements points 3–8 of the post-procedure follow-up. - Make targets: diagram-lookup, diagram-validate-spec, diagram-spec-audit, archcenter-descriptions-refresh, archcenter-smoke. Promotes the new tools out of "remember the CLI" into discoverable build-automation. - .gitea/workflows/diagram-validators.yaml — runs spec validator across every absolute_layout spec on push/PR, plus a drawio re-render smoke test on the canonical example. Hard fail blocks merges. - archcenter_pattern_lookup --llm-rewrite — opt-in (ANTHROPIC_API_KEY env var, mirroring DRAWIO_EXE pattern). Rewrites natural-language queries into canonical OCI terminology before scoring; falls back to the original query on any error. - diagram_spec_validator — documents the policy on legacy archcenter reproductions: LABEL_NEAR_PARENT_EDGE stays a warning so verbatim reconstructions don't lose pixel-fidelity from cosmetic edits. - README — documents the asset refresh procedure (downloader + description fetcher are idempotent, refresh quarterly). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-25 21:37:28 -03:00
parent b30a4f0d32
commit e5a1959585
5 changed files with 193 additions and 1 deletions
--- a/.gitea/workflows/diagram-validators.yaml
+++ b/.gitea/workflows/diagram-validators.yaml
@@ -0,0 +1,75 @@
+# Diagram-validator gate.
+#
+# Runs on every push and PR. Two checks:
+#
+#   1. diagram-spec-audit — every absolute_layout spec under examples/
+#      passes tools/diagram_spec_validator.py (geometry rules:
+#      CONTAINER_TOO_THIN, CONTAINER_PADDING_VIOLATION,
+#      LABEL_OVERFLOW_PARENT). Hard fail surfaces the regressions
+#      Diego flagged on the MySQL HeatWave HA example.
+#
+#   2. drawio-validators-smoke — re-renders the canonical example with
+#      tools/oci_diagram_gen.py so the post-render
+#      drawio_visual_validator runs (font sizes, dangling edges,
+#      duplicate ids, off-canvas geometry).
+#
+# Failures block the merge so a busted geometry never ships.
+
+name: Diagram validators
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - 'examples/**'
+      - 'kb/diagram/**'
+      - 'tools/diagram_spec_validator.py'
+      - 'tools/drawio_visual_validator.py'
+      - 'tools/oci_diagram_gen.py'
+      - 'tools/oci_pptx_diagram_gen.py'
+  pull_request:
+    branches: [main]
+  workflow_dispatch:
+
+jobs:
+  diagram-validators:
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Python
+        run: |
+          apt-get update -qq
+          apt-get install -y --no-install-recommends python3 python3-pip python3-yaml
+          python3 --version
+
+      - name: Install minimal deps
+        run: |
+          python3 -m pip install --quiet --upgrade pip
+          python3 -m pip install --quiet drawpyo lxml pyyaml
+
+      - name: Spec validator audit (every absolute_layout spec)
+        run: |
+          set -e
+          total=0; failed=0
+          for f in $(find examples -name 'diagram-spec.yaml' -o -name '*-diagram-spec.yaml'); do
+            total=$((total+1))
+            if ! python3 tools/diagram_spec_validator.py --spec "$f" --strict; then
+              failed=$((failed+1))
+            fi
+          done
+          echo "Validated $total spec(s); $failed hard-failed."
+          test $failed -eq 0
+
+      - name: Re-render canonical example (drawio post-render validator)
+        run: |
+          if [ -f examples/aws-app-mysql-heatwave-ha/specs/diagram-spec.yaml ]; then
+            python3 tools/oci_diagram_gen.py \
+              --spec examples/aws-app-mysql-heatwave-ha/specs/diagram-spec.yaml \
+              --output /tmp/aws-app-mysql-heatwave-ha.drawio --strict
+          else
+            echo "Canonical example missing — skipping drawio smoke test"
+          fi
--- a/31
+++ b/31
@@ -1,6 +1,6 @@
 # OCI Deal Accelerator — Build Automation

-.PHONY: help install test validate example diagram deck full clean lint codex-package update-icons freshness freshness-refresh sync-skill sku-discover pptx-icons-refresh archcenter-benchmark-20
+.PHONY: help install test validate example diagram deck full clean lint codex-package update-icons freshness freshness-refresh sync-skill sku-discover pptx-icons-refresh archcenter-benchmark-20 diagram-lookup diagram-validate-spec archcenter-descriptions-refresh diagram-spec-audit archcenter-smoke

 # Use venv if present, otherwise find best available python3
 ifneq (,$(wildcard .venv/bin/python))
@@ -83,6 +83,35 @@ archcenter-benchmark-20: ## Run 20-case Oracle Architecture Center native benchm
 		--fidelity-threshold 0.90 \
 		--output-root examples/eval-2026-04-25-archcenter-native-20-v8

+diagram-lookup: ## Find Architecture Center reference patterns. Usage: make diagram-lookup QUERY="mysql heatwave HA"
+	@if [ -z "$(QUERY)" ]; then echo "Usage: make diagram-lookup QUERY=\"<topology>\""; exit 1; fi
+	$(PYTHON) tools/archcenter_pattern_lookup.py "$(QUERY)" --top $${TOP:-5}
+
+diagram-validate-spec: ## Validate a diagram spec's geometry. Usage: make diagram-validate-spec SPEC=path/to/spec.yaml
+	@if [ -z "$(SPEC)" ]; then echo "Usage: make diagram-validate-spec SPEC=<path>"; exit 1; fi
+	$(PYTHON) tools/diagram_spec_validator.py --spec "$(SPEC)" --strict
+
+diagram-spec-audit: ## Run spec validator across every diagram-spec in examples/, print pass/fail summary
+	@echo "Auditing diagram specs under examples/..."
+	@total=0; failed=0; \
+	for f in $$(find examples -name 'diagram-spec.yaml' -o -name '*-diagram-spec.yaml'); do \
+	  total=$$((total+1)); \
+	  $(PYTHON) tools/diagram_spec_validator.py --spec "$$f" --strict >/dev/null 2>&1 || { \
+	    failed=$$((failed+1)); echo "  FAIL  $$f"; \
+	  }; \
+	done; \
+	echo "Audited $$total spec(s); $$failed failed."
+
+archcenter-descriptions-refresh: ## Re-fetch _description.md for every Architecture Center entry
+	$(PYTHON) tools/archcenter_description_fetcher.py --limit 200 --sleep 0.5
+
+archcenter-smoke: ## 3-case Architecture Center reconstruction smoke test (CI-friendly)
+	$(PYTHON) tools/oci_archcenter_batch.py \
+		--limit 3 \
+		--threshold 0.78 \
+		--fidelity-threshold 0.85 \
+		--output-root tmp/archcenter-smoke
+
 clean: ## Remove generated output files
 	rm -f examples/sample-output/*.drawio
 	rm -f examples/sample-output/*.pptx
--- a/README.md
+++ b/README.md
@@ -178,8 +178,13 @@ python tools/archcenter_pattern_lookup.py "fastconnect exacs cross region" --top

 # Re-fetch description text after a catalog refresh
 python tools/archcenter_description_fetcher.py --limit 200
+
+# Re-download cached drawio/svg/png assets after a catalog refresh
+python tools/archcenter_zip_downloader.py    # idempotent; skips slugs whose folder already has a .drawio
 ```

+The cached assets under `kb/diagram/assets/archcenter-refs/` (~83MB) are committed so the skill works offline. Refresh quarterly or when `refresh_arch_catalog.py --whats-new` adds entries — both downloader and description fetcher are idempotent and only fetch what's missing.
+
 ### KB Health & Freshness

 The KB is automatically monitored for staleness and broken links:
--- a/tools/archcenter_pattern_lookup.py
+++ b/tools/archcenter_pattern_lookup.py
@@ -33,7 +33,12 @@ as the basis for your absolute_layout authoring.
 from __future__ import annotations

 import argparse
+import json
+import os
 import re
+import sys
+import urllib.error
+import urllib.request
 from pathlib import Path

 import yaml
@@ -100,6 +105,60 @@ def _expand_query_tokens(tokens: set[str]) -> set[str]:
    return expanded


+def _llm_rewrite_query(query: str) -> str:
+    """Rewrite ``query`` into canonical OCI/Oracle terminology via the
+    Anthropic API.
+
+    Opt-in: requires ``ANTHROPIC_API_KEY`` in the environment. Returns
+    the original query unchanged on any error or when the key is
+    missing — never fails the lookup. The synonym table covers the
+    common abbreviations; this hook is for natural-language queries
+    where rule-based expansion alone misses the intent.
+    """
+    api_key = os.environ.get("ANTHROPIC_API_KEY")
+    if not api_key:
+        print("[lookup] --llm-rewrite skipped: ANTHROPIC_API_KEY not set",
+              file=sys.stderr)
+        return query
+    model = os.environ.get("ANTHROPIC_MODEL", "claude-haiku-4-5-20251001")
+    payload = {
+        "model": model,
+        "max_tokens": 120,
+        "system": (
+            "Rewrite the user's free-text architecture query into a "
+            "concise list of canonical OCI and cloud terms. Expand "
+            "abbreviations (LB → load balancer, ADG → active data "
+            "guard, AD → availability domain, etc.). Do NOT add "
+            "topology elements the user did not request. Output ONLY "
+            "the rewritten query as a single space-separated phrase, "
+            "no preface, no quotes."
+        ),
+        "messages": [{"role": "user", "content": query}],
+    }
+    req = urllib.request.Request(
+        "https://api.anthropic.com/v1/messages",
+        data=json.dumps(payload).encode("utf-8"),
+        headers={
+            "content-type": "application/json",
+            "x-api-key": api_key,
+            "anthropic-version": "2023-06-01",
+        },
+    )
+    try:
+        with urllib.request.urlopen(req, timeout=15) as resp:
+            body = json.loads(resp.read().decode("utf-8"))
+    except (urllib.error.URLError, urllib.error.HTTPError, TimeoutError, ValueError) as exc:
+        print(f"[lookup] --llm-rewrite failed: {exc}; using original query",
+              file=sys.stderr)
+        return query
+    blocks = body.get("content") or []
+    text = " ".join(b.get("text", "") for b in blocks if b.get("type") == "text").strip()
+    if not text:
+        return query
+    print(f"[lookup] llm-rewrote: {query!r} → {text!r}", file=sys.stderr)
+    return text
+
+
 def _description_text(entry: dict) -> str:
    """Read the cached `_description.md` body if present.

@@ -251,6 +310,11 @@ def main() -> None:
    parser.add_argument("--no-synonyms", action="store_true",
                        help="Disable query-token expansion via "
                        "kb/architecture-center/synonyms.yaml.")
+    parser.add_argument("--llm-rewrite", action="store_true",
+                        help="Rewrite the query via the Anthropic API "
+                        "before scoring (opt-in; requires "
+                        "ANTHROPIC_API_KEY). Useful for natural-language "
+                        "queries the synonym table doesn't cover.")
    args = parser.parse_args()

    queries: list[str] = list(args.queries)
@@ -260,6 +324,8 @@ def main() -> None:
        parser.error("Provide a positional query or --queries.")

    expand = not args.no_synonyms
+    if args.llm_rewrite:
+        queries = [_llm_rewrite_query(q) for q in queries]
    if args.format == "yaml":
        bundle = {q: lookup(q, top=args.top, expand_synonyms=expand) for q in queries}
        print(yaml.safe_dump(bundle if len(queries) > 1 else bundle[queries[0]],
--- a/tools/diagram_spec_validator.py
+++ b/tools/diagram_spec_validator.py
@@ -29,6 +29,23 @@ Returns a dict with:
  - issues: [{"severity", "code", "message", "id"}]

 Severity ``error`` → renderers raise. ``warn`` → printed to stderr.
+
+Policy on legacy archcenter reproductions
+-----------------------------------------
+Specs under ``examples/eval-*-archcenter-native-*/`` are auto-generated
+reconstructions of Oracle Architecture Center references. Oracle's own
+geometry sometimes places labels 2-5px from container edges, and the
+reproductions mirror those coordinates verbatim to keep the perceptual
+diff against the canonical PNGs above the fidelity threshold. Touching
+them to silence ``LABEL_NEAR_PARENT_EDGE`` (severity=warn) would risk
+the pixel-fidelity scores tracked by ``archcenter_batch_driver``.
+
+The validator therefore distinguishes:
+
+  - ``LABEL_OVERFLOW_PARENT`` (error, blocks render): label bottom is
+    OUTSIDE the parent — true regression.
+  - ``LABEL_NEAR_PARENT_EDGE`` (warn, never blocks): label sits within
+    6px but still inside — common in Oracle's own ref archs.
 """
 from __future__ import annotations