SPEC-006 — Evals

Scope

The layer of Evals executa Evaluation online, Evaluation offline, regressão, Certification and publicação of Metrics. Ela padroniza Validation of Agents, prompts, tools, responses and guardrails.

Components

Componente	Responsibility
Online Judges	Evaluation during a Execution.
Offline Evaluator	Evaluation batch of conversas.
Dataset Runner	Execution of datasets versionados.
Regression Runner	Comparação between versions.
Certification Suite	Validation técnica and funcional.
Metrics Engine	Cálculo of Metrics.
Persistence	Persistence of runs and itens.
Exporter	Exportação TXT.GZ/JSON/HTML.
Publisher	Publicação of scores no Langfuse.

Flow Offline

flowchart TD
    A[Start EvaluationRun] --> B[Collect Conversations]
    B --> C[Normalize Items]
    C --> D[Run Judges]
    D --> E[Calculate Metrics]
    E --> F[Persist Results]
    F --> G[Export Reports]
    G --> H[Publish Scores]
    H --> I[Complete Run]

EvaluationRun

{
  "run_id": "eval-20260619-001",
  "agent_id": "telecom_contas",
  "source": "langfuse",
  "period_start": "2026-06-18T00:00:00Z",
  "period_end": "2026-06-19T00:00:00Z",
  "status": "running",
  "limit": 500,
  "metadata": {
    "profile": "judge",
    "dataset": "production-sample"
  }
}

EvaluationItem

{
  "conversation_id": "default:telecom_contas:session-001",
  "trace_id": "trace-001",
  "agent_id": "telecom_contas",
  "input": "Quero consultar minha fatura",
  "output": "Sua fatura está aberta...",
  "evidence": {
    "mcp_results": [],
    "rag_context": ""
  },
  "scores": {
    "quality": 0.86,
    "groundedness": 0.78,
    "safety": 1.0,
    "resolution": 0.91
  },
  "findings": []
}

Metrics

metric	Description	Faixa
`quality`	Clareza, completude and utilidade.	0–1
`groundedness`	Aderência evidências MCP/RAG.	0–1
`safety`	compliance of Security.	0–1
`resolution`	Capacity of resolve a intent.	0–1
`tool_correctness`	Uso correct of tools.	0–1
`policy_compliance`	Aderência Rules of Domain.	0–1

Dataset

dataset:
  name: telecom_contas_billing
  version: 1.0.0
  items:
    - id: billing-001
      input: "Quero consultar minha fatura"
      business_context:
        customer_key: "11999999999"
        contract_key: "3000131180"
      expected:
        route: billing_agent
        tools:
          - consultar_fatura
        min_scores:
          quality: 0.75
          groundedness: 0.70
          safety: 1.0

Judges

judges:
  - name: response_quality
    enabled: true
    threshold: 0.7
    profile: judge

  - name: groundedness
    enabled: true
    threshold: 0.6
    profile: judge

  - name: safety
    enabled: true
    threshold: 1.0
    profile: judge

CLI

af-evaluator run \
  --agent-id telecom_contas \
  --source langfuse \
  --period-start 2026-06-18T00:00:00Z \
  --period-end 2026-06-19T00:00:00Z \
  --limit 500

API

Method	Endpoint	Description
`POST`	`/evaluation/runs`	Cria run.
`GET`	`/evaluation/runs/{run_id}`	Consulta run.
`GET`	`/evaluation/runs/{run_id}/items`	Lista itens.
`POST`	`/evaluation/datasets/{name}/run`	Executa dataset.
`GET`	`/health`	Health check.

Persistence

table	Conteúdo
`EVAL_RUNS`	Runs executadas.
`EVAL_ITEMS`	Conversas avaliadas.
`EVAL_SCORES`	Scores por metric.
`EVAL_FINDINGS`	Achados.
`EVAL_EXPORTS`	files exportados.

Certification

A Certification Suite valida:

endpoints of health;
GatewayRequest;
Routing;
MCP tools;
guardrails;
judges;
Memory;
checkpoint;
Langfuse/OTEL;
datasets minimum;
evidências JSON/HTML.

Events

Evento	Description
`eval.run.started`	Run iniciada.
`eval.item.completed`	Item avaliado.
`eval.run.completed`	Run concluída.
`eval.run.failed`	Run failed.
`eval.score.published`	Score publicado.

Requirements No Funcionais

Categoria	Requirement
Availability	Components deployáveis expõem `/health` and `/ready`.
Escalabilidade	Apps stateless escalam horizontalmente. State conversational fica em repositórios external.
Security	Segredos are fornecidos por secret store or Kubernetes Secrets.
Observability	Logs, Metrics and traces usam correlation por request_id, trace_id, session_id, tenant_id and agent_id.
Auditabilidade	Decisions of rota, guardrail, judge, MCP and LLM are rastreáveis.
Portabilidade	Execution suportada em local, Docker Compose and Kubernetes/OKE.
Configuration	Comportamento variável is controlado por `.env` and YAML versionado.

Acceptance criteria

Evaluator executa runs por período/Agent.
Langfuse is source suportada.
Datasets are versionados.
LLM Judges usam profile judge.
Scores are persistidos.
TXT.GZ/JSON/HTML are exportáveis.
Scores Can be publicados no Langfuse.
Certification Suite gera evidências.
Metrics mínimas are padronizadas.
Failures permitem retomada por checkpoint of run.

Glossário

Termo	Definição
Agent Platform	Plataforma composta por runtime, gateways, evaluator, templates, contracts and Components operational.
Agent Framework	Biblioteca/core reutilizável with contracts, guardrails, judges, Memory, telemetria, providers and utilitários.
Agent Runtime	Motor of Execution of Agents based on LangGraph, State, Session, Memory, checkpoints, Routing and ciclo of vida.
Agent Gateway	Aplicação deployável of Input, Routing and orquestração between backends/Agents.
Channel Gateway	Aplicação or módulo of normalização of payloads of canais for GatewayRequest.
AI Gateway	Aplicação of Governance, Routing and abstração of Calls LLM/embedding.
MCP Gateway	Aplicação of Governance and Routing of tools MCP.
Evaluator	layer of Evaluation online/offline, regressão and Certification.
Business Context	Conjunto of chaves canônicas of negócio: customer_key, contract_key, interaction_key, account_key, resource_key and session_key.

6.2 KiB Raw Blame History Unescape Escape