# SPEC-019 — Evaluation and Certification Framework

## Agent Platform OCI

Version: 1.0.0


---

## Reading guide

each SPEC is organizada for servir tanto como Contract arquitetural quanto como guia practical of adoção.

The document structure is:

1. Concept.
2. Problem addressed.
3. When to use.
4. When not to use.
5. Architecture.
6. Implementation.
7. Examples.
8. Common errors.
9. Acceptance criteria.

---


# 1. Concept

Evaluation mede quality and comportamento. Certification valida prontidão técnica and funcional.

Evaluator responde:

```text
O agente respondeu bem?
A resposta está fundamentada?
A tool certa foi chamada?
Houve regressão?
```

Certification responde:

```text
O agente está pronto para rodar?
Endpoints funcionam?
MCP funciona?
Guardrails funcionam?
Observabilidade funciona?
```

# 2. Architecture

```mermaid
flowchart LR
    Runtime[Runtime] --> LF[Langfuse]
    LF --> Eval[Offline Evaluator]
    Dataset[Datasets] --> Eval
    Eval --> Scores[Scores]
    Eval --> Reports[Reports]
    Cert[Certification Suite] --> Runtime
    Cert --> Evidence[Evidences]
```

# 3. Metrics

| metric | Description |
| --- | --- |
| quality | Clareza, completude and utilidade. |
| groundedness | Aderência evidências MCP/RAG. |
| safety | compliance of Security. |
| resolution | Resolve a intent. |
| tool_correctness | Uses tools corretas. |
| route_accuracy | Rota/intent corretas. |
| policy_compliance | Aderência à Policy of Domain. |


# 4. Dataset

```yaml
dataset:
  name: telecom_contas_regression
  version: 1.0.0
  items:
    - id: billing-001
      input: "Quero consultar minha fatura"
      business_context:
        customer_key: "11999999999"
        contract_key: "3000131180"
      expected:
        route: billing_agent
        tools:
          - consultar_fatura
        min_scores:
          quality: 0.75
          groundedness: 0.70
```

# 5. EvaluationRun

```json
{
  "run_id": "eval-001",
  "agent_id": "telecom_contas",
  "source": "langfuse",
  "period_start": "2026-06-18T00:00:00Z",
  "period_end": "2026-06-19T00:00:00Z",
  "status": "running"
}
```

# 6. CLI

```bash
af-evaluator run   --agent-id telecom_contas   --dataset datasets/telecom_contas.yaml
```

# 7. Certification

Valida:

- health;
- GatewayRequest;
- routing;
- identity;
- MCP;
- RAG;
- guardrails;
- judges;
- memory;
- checkpoint;
- Langfuse;
- OTEL.

# 8. Evidências

- JSON;
- HTML;
- TXT.GZ legado;
- scores Langfuse;
- logs;
- traces;
- screenshots when aplicável.

# 9. Common errors

| Error | Impact | Correction |
| --- | --- | --- |
| Dataset só with casos felizes | Baixa coverage. | include negativos and bordas. |
| Evaluator without baseline | without comparação. | Registrar baseline. |
| Certification without MCP real/mock | integration No validada. | create tool test. |
| Judge without threshold | without criterion objective. | Definir threshold. |


# 10. Acceptance criteria

- [ ] Dataset versionado.
- [ ] Evaluator executado.
- [ ] Scores persistidos.
- [ ] Certification executada.
- [ ] Relatórios gerados.
- [ ] Thresholds definidos.
- [ ] Casos negativos incluídos.
- [ ] Scores publicados when aplicável.