agent_framework_oci/specs/en/SPEC-019-Evaluation-and-Certification-Framework.md

SPEC-019 — Evaluation and Certification Framework

Agent Platform OCI

Version: 1.0.0

---

Reading guide

each SPEC is organizada for servir tanto como Contract arquitetural quanto como guia practical of adoção.

The document structure is:

1. Concept.
2. Problem addressed.
3. When to use.
4. When not to use.
5. Architecture.
6. Implementation.
7. Examples.
8. Common errors.
9. Acceptance criteria.

---

1. Concept

Evaluation mede quality and comportamento. Certification valida prontidão técnica and funcional.

Evaluator responde:

O agente respondeu bem?
A resposta está fundamentada?
A tool certa foi chamada?
Houve regressão?

Certification responde:

O agente está pronto para rodar?
Endpoints funcionam?
MCP funciona?
Guardrails funcionam?
Observabilidade funciona?

2. Architecture

flowchart LR
    Runtime[Runtime] --> LF[Langfuse]
    LF --> Eval[Offline Evaluator]
    Dataset[Datasets] --> Eval
    Eval --> Scores[Scores]
    Eval --> Reports[Reports]
    Cert[Certification Suite] --> Runtime
    Cert --> Evidence[Evidences]

3. Metrics

metric	Description
quality	Clareza, completude and utilidade.
groundedness	Aderência evidências MCP/RAG.
safety	compliance of Security.
resolution	Resolve a intent.
tool_correctness	Uses tools corretas.
route_accuracy	Rota/intent corretas.
policy_compliance	Aderência à Policy of Domain.

4. Dataset

dataset:
  name: telecom_contas_regression
  version: 1.0.0
  items:
    - id: billing-001
      input: "Quero consultar minha fatura"
      business_context:
        customer_key: "11999999999"
        contract_key: "3000131180"
      expected:
        route: billing_agent
        tools:
          - consultar_fatura
        min_scores:
          quality: 0.75
          groundedness: 0.70

5. EvaluationRun

{
  "run_id": "eval-001",
  "agent_id": "telecom_contas",
  "source": "langfuse",
  "period_start": "2026-06-18T00:00:00Z",
  "period_end": "2026-06-19T00:00:00Z",
  "status": "running"
}

6. CLI

af-evaluator run   --agent-id telecom_contas   --dataset datasets/telecom_contas.yaml

7. Certification

Valida:

• health;
• GatewayRequest;
• routing;
• identity;
• MCP;
• RAG;
• guardrails;
• judges;
• memory;
• checkpoint;
• Langfuse;
• OTEL.

8. Evidências

• JSON;
• HTML;
• TXT.GZ legado;
• scores Langfuse;
• logs;
• traces;
• screenshots when aplicável.

9. Common errors

Error	Impact	Correction
Dataset só with casos felizes	Baixa coverage.	include negativos and bordas.
Evaluator without baseline	without comparação.	Registrar baseline.
Certification without MCP real/mock	integration No validada.	create tool test.
Judge without threshold	without criterion objective.	Definir threshold.

10. Acceptance criteria

• Dataset versionado.
• Evaluator executado.
• Scores persistidos.
• Certification executada.
• Relatórios gerados.
• Thresholds definidos.
• Casos negativos incluídos.
• Scores publicados when aplicável.