167 lines
3.0 KiB
Markdown
167 lines
3.0 KiB
Markdown
# SPEC-020 — Operational Readiness and SRE Model
|
|
|
|
## Agent Platform OCI
|
|
|
|
Version: 1.0.0
|
|
|
|
|
|
---
|
|
|
|
## Reading guide
|
|
|
|
each SPEC is organizada for servir tanto como Contract arquitetural quanto como guia practical of adoção.
|
|
|
|
The document structure is:
|
|
|
|
1. Concept.
|
|
2. Problem addressed.
|
|
3. When to use.
|
|
4. When not to use.
|
|
5. Architecture.
|
|
6. Implementation.
|
|
7. Examples.
|
|
8. Common errors.
|
|
9. Acceptance criteria.
|
|
|
|
---
|
|
|
|
|
|
# 1. Concept
|
|
|
|
Operational readiness defines the minimum requirements to run the Agent Platform OCI in production with reliability, observability, incident response capabilities, and recovery.
|
|
|
|
# 2. Managed components
|
|
|
|
- Agent Gateway;
|
|
- Channel Gateway;
|
|
- Agent Runtime;
|
|
- AI Gateway;
|
|
- MCP Gateway;
|
|
- MCP Servers;
|
|
- Evaluator;
|
|
- bancos/repositórios;
|
|
- Langfuse/OTEL;
|
|
- Redis/Mongo/ADB when usados.
|
|
|
|
# 3. Health and readiness
|
|
|
|
Minimum endpoints:
|
|
|
|
```text
|
|
GET /health
|
|
GET /ready
|
|
GET /version
|
|
```
|
|
|
|
# 4. SLOs
|
|
|
|
| Componente | Latency | Availability |
|
|
| --- | --- | --- |
|
|
| Agent Gateway | p95 < 1s | 99.5% |
|
|
| Agent Runtime | p95 < 5s | 99.0% |
|
|
| AI Gateway | p95 < 10s | 99.0% |
|
|
| MCP Gateway | p95 < 2s | 99.0% |
|
|
| Evaluator | janela batch | Execution diária |
|
|
|
|
|
|
# 5. Metrics
|
|
|
|
- requests_total;
|
|
- request_latency_ms;
|
|
- errors_total;
|
|
- active_sessions;
|
|
- llm_tokens_total;
|
|
- llm_cost_estimated;
|
|
- mcp_tool_calls_total;
|
|
- guardrail_blocks_total;
|
|
- judge_scores;
|
|
- evaluator_scores.
|
|
|
|
# 6. Dashboards
|
|
|
|
Minimum dashboards:
|
|
|
|
- Platform Overview;
|
|
- Runtime;
|
|
- Gateway;
|
|
- AI Gateway;
|
|
- MCP Gateway;
|
|
- Guardrails;
|
|
- Evaluator;
|
|
- Cost/Usage;
|
|
- Incidents.
|
|
|
|
# 7. Alerts
|
|
|
|
| Alert | Condition |
|
|
| --- | --- |
|
|
| HighErrorRate | 5xx above of the Limit. |
|
|
| LatencySLOBreach | p95 above of the SLO. |
|
|
| LLMProviderDown | Failures consecutive no provider. |
|
|
| MCPTimeoutSpike | increase of timeout MCP. |
|
|
| GuardrailSpike | increase abnormal of Blocks. |
|
|
| EvaluatorFailed | Run failed. |
|
|
|
|
|
|
# 8. Runbooks
|
|
|
|
A runbook must contain:
|
|
|
|
- symptom;
|
|
- Impact;
|
|
- queries;
|
|
- dashboards;
|
|
- logs;
|
|
- actions;
|
|
- rollback;
|
|
- escalation.
|
|
|
|
# 9. Incident management
|
|
|
|
Flow:
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
Detect[Detect] --> Triage[Triage]
|
|
Triage --> Mitigate[Mitigate]
|
|
Mitigate --> Recover[Recover]
|
|
Recover --> Postmortem[Postmortem]
|
|
```
|
|
|
|
# 10. Capacity
|
|
|
|
Assess:
|
|
|
|
- QPS;
|
|
- Sessions concurrent;
|
|
- tokens/minuto;
|
|
- Calls MCP/minuto;
|
|
- Latency of provider;
|
|
- uso of Memory;
|
|
- storage of checkpoints.
|
|
|
|
# 11. Common errors
|
|
|
|
| Error | Impact | Correction |
|
|
| --- | --- | --- |
|
|
| without readiness | traffic before of app estar ready. | implement /ready. |
|
|
| without Alerts MCP | Failure silenciosa. | create Alerts por tool. |
|
|
| without runbook | MTTR high. | create runbooks por incidente. |
|
|
| without Cost LLM | without control financial. | Registrar tokens/Costs. |
|
|
|
|
|
|
# 12. Production readiness checklist
|
|
|
|
- [ ] Health checks ativos.
|
|
- [ ] Readiness checks ativos.
|
|
- [ ] Structured logs.
|
|
- [ ] Metrics exportadas.
|
|
- [ ] Exported traces.
|
|
- [ ] Dashboards created.
|
|
- [ ] Configured alerts.
|
|
- [ ] Runbooks disponíveis.
|
|
- [ ] Rollback validado.
|
|
- [ ] Defined SLOs.
|
|
- [ ] Estimated capacity.
|
|
- [ ] Incident process definido.
|