mirror of
https://github.com/hoshikawa2/agent_platform_oci.git
synced 2026-11-06 11:24:13 +00:00
165 lines
3.1 KiB
Markdown
165 lines
3.1 KiB
Markdown
# SPEC-019 — Evaluation and Certification Framework
|
|
|
|
## Agent Platform OCI
|
|
|
|
Version: 1.0.0
|
|
|
|
|
|
---
|
|
|
|
## Reading guide
|
|
|
|
each SPEC is organizada for servir tanto como Contract arquitetural quanto como guia practical of adoção.
|
|
|
|
The document structure is:
|
|
|
|
1. Concept.
|
|
2. Problem addressed.
|
|
3. When to use.
|
|
4. When not to use.
|
|
5. Architecture.
|
|
6. Implementation.
|
|
7. Examples.
|
|
8. Common errors.
|
|
9. Acceptance criteria.
|
|
|
|
---
|
|
|
|
|
|
# 1. Concept
|
|
|
|
Evaluation mede quality and comportamento. Certification valida prontidão técnica and funcional.
|
|
|
|
Evaluator responde:
|
|
|
|
```text
|
|
O agente respondeu bem?
|
|
A resposta está fundamentada?
|
|
A tool certa foi chamada?
|
|
Houve regressão?
|
|
```
|
|
|
|
Certification responde:
|
|
|
|
```text
|
|
O agente está pronto para rodar?
|
|
Endpoints funcionam?
|
|
MCP funciona?
|
|
Guardrails funcionam?
|
|
Observabilidade funciona?
|
|
```
|
|
|
|
# 2. Architecture
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
Runtime[Runtime] --> LF[Langfuse]
|
|
LF --> Eval[Offline Evaluator]
|
|
Dataset[Datasets] --> Eval
|
|
Eval --> Scores[Scores]
|
|
Eval --> Reports[Reports]
|
|
Cert[Certification Suite] --> Runtime
|
|
Cert --> Evidence[Evidences]
|
|
```
|
|
|
|
# 3. Metrics
|
|
|
|
| metric | Description |
|
|
| --- | --- |
|
|
| quality | Clareza, completude and utilidade. |
|
|
| groundedness | Aderência evidências MCP/RAG. |
|
|
| safety | compliance of Security. |
|
|
| resolution | Resolve a intent. |
|
|
| tool_correctness | Uses tools corretas. |
|
|
| route_accuracy | Rota/intent corretas. |
|
|
| policy_compliance | Aderência à Policy of Domain. |
|
|
|
|
|
|
# 4. Dataset
|
|
|
|
```yaml
|
|
dataset:
|
|
name: telecom_contas_regression
|
|
version: 1.0.0
|
|
items:
|
|
- id: billing-001
|
|
input: "Quero consultar minha fatura"
|
|
business_context:
|
|
customer_key: "11999999999"
|
|
contract_key: "3000131180"
|
|
expected:
|
|
route: billing_agent
|
|
tools:
|
|
- consultar_fatura
|
|
min_scores:
|
|
quality: 0.75
|
|
groundedness: 0.70
|
|
```
|
|
|
|
# 5. EvaluationRun
|
|
|
|
```json
|
|
{
|
|
"run_id": "eval-001",
|
|
"agent_id": "telecom_contas",
|
|
"source": "langfuse",
|
|
"period_start": "2026-06-18T00:00:00Z",
|
|
"period_end": "2026-06-19T00:00:00Z",
|
|
"status": "running"
|
|
}
|
|
```
|
|
|
|
# 6. CLI
|
|
|
|
```bash
|
|
af-evaluator run --agent-id telecom_contas --dataset datasets/telecom_contas.yaml
|
|
```
|
|
|
|
# 7. Certification
|
|
|
|
Valida:
|
|
|
|
- health;
|
|
- GatewayRequest;
|
|
- routing;
|
|
- identity;
|
|
- MCP;
|
|
- RAG;
|
|
- guardrails;
|
|
- judges;
|
|
- memory;
|
|
- checkpoint;
|
|
- Langfuse;
|
|
- OTEL.
|
|
|
|
# 8. Evidências
|
|
|
|
- JSON;
|
|
- HTML;
|
|
- TXT.GZ legado;
|
|
- scores Langfuse;
|
|
- logs;
|
|
- traces;
|
|
- screenshots when aplicável.
|
|
|
|
# 9. Common errors
|
|
|
|
| Error | Impact | Correction |
|
|
| --- | --- | --- |
|
|
| Dataset só with casos felizes | Baixa coverage. | include negativos and bordas. |
|
|
| Evaluator without baseline | without comparação. | Registrar baseline. |
|
|
| Certification without MCP real/mock | integration No validada. | create tool test. |
|
|
| Judge without threshold | without criterion objective. | Definir threshold. |
|
|
|
|
|
|
# 10. Acceptance criteria
|
|
|
|
- [ ] Dataset versionado.
|
|
- [ ] Evaluator executado.
|
|
- [ ] Scores persistidos.
|
|
- [ ] Certification executada.
|
|
- [ ] Relatórios gerados.
|
|
- [ ] Thresholds definidos.
|
|
- [ ] Casos negativos incluídos.
|
|
- [ ] Scores publicados when aplicável.
|