# SPEC-006 — Evals

## Scope

The layer of Evals executa Evaluation online, Evaluation offline, regressão, Certification and publicação of Metrics. Ela padroniza Validation of Agents, prompts, tools, responses and guardrails.

## Components

| Componente | Responsibility |
|---|---|
| Online Judges | Evaluation during a Execution. |
| Offline Evaluator | Evaluation batch of conversas. |
| Dataset Runner | Execution of datasets versionados. |
| Regression Runner | Comparação between versions. |
| Certification Suite | Validation técnica and funcional. |
| Metrics Engine | Cálculo of Metrics. |
| Persistence | Persistence of runs and itens. |
| Exporter | Exportação TXT.GZ/JSON/HTML. |
| Publisher | Publicação of scores no Langfuse. |

## Flow Offline

```mermaid
flowchart TD
    A[Start EvaluationRun] --> B[Collect Conversations]
    B --> C[Normalize Items]
    C --> D[Run Judges]
    D --> E[Calculate Metrics]
    E --> F[Persist Results]
    F --> G[Export Reports]
    G --> H[Publish Scores]
    H --> I[Complete Run]
```

## EvaluationRun

```json
{
  "run_id": "eval-20260619-001",
  "agent_id": "telecom_contas",
  "source": "langfuse",
  "period_start": "2026-06-18T00:00:00Z",
  "period_end": "2026-06-19T00:00:00Z",
  "status": "running",
  "limit": 500,
  "metadata": {
    "profile": "judge",
    "dataset": "production-sample"
  }
}
```

## EvaluationItem

```json
{
  "conversation_id": "default:telecom_contas:session-001",
  "trace_id": "trace-001",
  "agent_id": "telecom_contas",
  "input": "Quero consultar minha fatura",
  "output": "Sua fatura está aberta...",
  "evidence": {
    "mcp_results": [],
    "rag_context": ""
  },
  "scores": {
    "quality": 0.86,
    "groundedness": 0.78,
    "safety": 1.0,
    "resolution": 0.91
  },
  "findings": []
}
```

## Metrics

| metric | Description | Faixa |
|---|---|---|
| `quality` | Clareza, completude and utilidade. | 0–1 |
| `groundedness` | Aderência evidências MCP/RAG. | 0–1 |
| `safety` | compliance of Security. | 0–1 |
| `resolution` | Capacity of resolve a intent. | 0–1 |
| `tool_correctness` | Uso correct of tools. | 0–1 |
| `policy_compliance` | Aderência Rules of Domain. | 0–1 |

## Dataset

```yaml
dataset:
  name: telecom_contas_billing
  version: 1.0.0
  items:
    - id: billing-001
      input: "Quero consultar minha fatura"
      business_context:
        customer_key: "11999999999"
        contract_key: "3000131180"
      expected:
        route: billing_agent
        tools:
          - consultar_fatura
        min_scores:
          quality: 0.75
          groundedness: 0.70
          safety: 1.0
```

## Judges

```yaml
judges:
  - name: response_quality
    enabled: true
    threshold: 0.7
    profile: judge

  - name: groundedness
    enabled: true
    threshold: 0.6
    profile: judge

  - name: safety
    enabled: true
    threshold: 1.0
    profile: judge
```

## CLI

```bash
af-evaluator run \
  --agent-id telecom_contas \
  --source langfuse \
  --period-start 2026-06-18T00:00:00Z \
  --period-end 2026-06-19T00:00:00Z \
  --limit 500
```

## API

| Method | Endpoint | Description |
|---|---|---|
| `POST` | `/evaluation/runs` | Cria run. |
| `GET` | `/evaluation/runs/{run_id}` | Consulta run. |
| `GET` | `/evaluation/runs/{run_id}/items` | Lista itens. |
| `POST` | `/evaluation/datasets/{name}/run` | Executa dataset. |
| `GET` | `/health` | Health check. |

## Persistence

| table | Conteúdo |
|---|---|
| `EVAL_RUNS` | Runs executadas. |
| `EVAL_ITEMS` | Conversas avaliadas. |
| `EVAL_SCORES` | Scores por metric. |
| `EVAL_FINDINGS` | Achados. |
| `EVAL_EXPORTS` | files exportados. |

## Certification

A Certification Suite valida:

- endpoints of health;
- GatewayRequest;
- Routing;
- MCP tools;
- guardrails;
- judges;
- Memory;
- checkpoint;
- Langfuse/OTEL;
- datasets minimum;
- evidências JSON/HTML.

## Events

| Evento | Description |
|---|---|
| `eval.run.started` | Run iniciada. |
| `eval.item.completed` | Item avaliado. |
| `eval.run.completed` | Run concluída. |
| `eval.run.failed` | Run failed. |
| `eval.score.published` | Score publicado. |


## Requirements No Funcionais

| Categoria | Requirement |
|---|---|
| Availability | Components deployáveis expõem `/health` and `/ready`. |
| Escalabilidade | Apps stateless escalam horizontalmente. State conversational fica em repositórios external. |
| Security | Segredos are fornecidos por secret store or Kubernetes Secrets. |
| Observability | Logs, Metrics and traces usam correlation por request_id, trace_id, session_id, tenant_id and agent_id. |
| Auditabilidade | Decisions of rota, guardrail, judge, MCP and LLM are rastreáveis. |
| Portabilidade | Execution suportada em local, Docker Compose and Kubernetes/OKE. |
| Configuration | Comportamento variável is controlado por `.env` and YAML versionado. |


## Acceptance criteria

- [ ] Evaluator executa runs por período/Agent.
- [ ] Langfuse is source suportada.
- [ ] Datasets are versionados.
- [ ] LLM Judges usam profile `judge`.
- [ ] Scores are persistidos.
- [ ] TXT.GZ/JSON/HTML are exportáveis.
- [ ] Scores Can be publicados no Langfuse.
- [ ] Certification Suite gera evidências.
- [ ] Metrics mínimas are padronizadas.
- [ ] Failures permitem retomada por checkpoint of run.


## Glossário

| Termo | Definição |
|---|---|
| Agent Platform | Plataforma composta por runtime, gateways, evaluator, templates, contracts and Components operational. |
| Agent Framework | Biblioteca/core reutilizável with contracts, guardrails, judges, Memory, telemetria, providers and utilitários. |
| Agent Runtime | Motor of Execution of Agents based on LangGraph, State, Session, Memory, checkpoints, Routing and ciclo of vida. |
| Agent Gateway | Aplicação deployável of Input, Routing and orquestração between backends/Agents. |
| Channel Gateway | Aplicação or módulo of normalização of payloads of canais for GatewayRequest. |
| AI Gateway | Aplicação of Governance, Routing and abstração of Calls LLM/embedding. |
| MCP Gateway | Aplicação of Governance and Routing of tools MCP. |
| Evaluator | layer of Evaluation online/offline, regressão and Certification. |
| Business Context | Conjunto of chaves canônicas of negócio: customer_key, contract_key, interaction_key, account_key, resource_key and session_key. |