fix: Spec Agent Runtime

specs/en/SPEC-019-Evaluation-and-Certification-Framework.md

@@ -0,0 +1,164 @@
+# SPEC-019 — Evaluation and Certification Framework
+
+## Agent Platform OCI
+
+Version: 1.0.0
+
+
+---
+
+## Reading guide
+
+each SPEC is organizada for servir tanto como Contract arquitetural quanto como guia practical of adoção.
+
+The document structure is:
+
+1. Concept.
+2. Problem addressed.
+3. When to use.
+4. When not to use.
+5. Architecture.
+6. Implementation.
+7. Examples.
+8. Common errors.
+9. Acceptance criteria.
+
+---
+
+
+# 1. Concept
+
+Evaluation mede quality and comportamento. Certification valida prontidão técnica and funcional.
+
+Evaluator responde:
+
+```text
+O agente respondeu bem?
+A resposta está fundamentada?
+A tool certa foi chamada?
+Houve regressão?
+```
+
+Certification responde:
+
+```text
+O agente está pronto para rodar?
+Endpoints funcionam?
+MCP funciona?
+Guardrails funcionam?
+Observabilidade funciona?
+```
+
+# 2. Architecture
+
+```mermaid
+flowchart LR
+    Runtime[Runtime] --> LF[Langfuse]
+    LF --> Eval[Offline Evaluator]
+    Dataset[Datasets] --> Eval
+    Eval --> Scores[Scores]
+    Eval --> Reports[Reports]
+    Cert[Certification Suite] --> Runtime
+    Cert --> Evidence[Evidences]
+```
+
+# 3. Metrics
+
+| metric | Description |
+| --- | --- |
+| quality | Clareza, completude and utilidade. |
+| groundedness | Aderência evidências MCP/RAG. |
+| safety | compliance of Security. |
+| resolution | Resolve a intent. |
+| tool_correctness | Uses tools corretas. |
+| route_accuracy | Rota/intent corretas. |
+| policy_compliance | Aderência à Policy of Domain. |
+
+
+# 4. Dataset
+
+```yaml
+dataset:
+  name: telecom_contas_regression
+  version: 1.0.0
+  items:
+    - id: billing-001
+      input: "Quero consultar minha fatura"
+      business_context:
+        customer_key: "11999999999"
+        contract_key: "3000131180"
+      expected:
+        route: billing_agent
+        tools:
+          - consultar_fatura
+        min_scores:
+          quality: 0.75
+          groundedness: 0.70
+```
+
+# 5. EvaluationRun
+
+```json
+{
+  "run_id": "eval-001",
+  "agent_id": "telecom_contas",
+  "source": "langfuse",
+  "period_start": "2026-06-18T00:00:00Z",
+  "period_end": "2026-06-19T00:00:00Z",
+  "status": "running"
+}
+```
+
+# 6. CLI
+
+```bash
+af-evaluator run   --agent-id telecom_contas   --dataset datasets/telecom_contas.yaml
+```
+
+# 7. Certification
+
+Valida:
+
+- health;
+- GatewayRequest;
+- routing;
+- identity;
+- MCP;
+- RAG;
+- guardrails;
+- judges;
+- memory;
+- checkpoint;
+- Langfuse;
+- OTEL.
+
+# 8. Evidências
+
+- JSON;
+- HTML;
+- TXT.GZ legado;
+- scores Langfuse;
+- logs;
+- traces;
+- screenshots when aplicável.
+
+# 9. Common errors
+
+| Error | Impact | Correction |
+| --- | --- | --- |
+| Dataset só with casos felizes | Baixa coverage. | include negativos and bordas. |
+| Evaluator without baseline | without comparação. | Registrar baseline. |
+| Certification without MCP real/mock | integration No validada. | create tool test. |
+| Judge without threshold | without criterion objective. | Definir threshold. |
+
+
+# 10. Acceptance criteria
+
+- [ ] Dataset versionado.
+- [ ] Evaluator executado.
+- [ ] Scores persistidos.
+- [ ] Certification executada.
+- [ ] Relatórios gerados.
+- [ ] Thresholds definidos.
+- [ ] Casos negativos incluídos.
+- [ ] Scores publicados when aplicável.