agent_framework_llm_as_a_judge/README.en-US.md

# Agent Framework OCI Evaluator — LLM-as-a-Judge

**Language:** English | [Português](README.md)

## 1. Overview

The **Agent Framework OCI Evaluator** is the batch component responsible for asynchronously evaluating the quality of conversations produced by agents built on the Agent Framework OCI.

The service reads sessions and traces from **Langfuse v3**, deterministically samples sessions, calculates quality metrics with an **LLM-as-a-Judge**, persists state and results in **Oracle Autonomous Database**, generates the official `.TXT.GZ` artifact, publishes that artifact to **Google Cloud Storage (GCS)**, and audits the evaluator job itself in Langfuse.

It is designed for **D+1 execution** as a **Kubernetes CronJob**, with no runtime HTTP/gRPC endpoint.

> **Recommended deployment model:** one evaluator instance per agent/squad. The code is generic and supports multiple agents in configuration, but in production each deployment should normally enable only the agent owned by that squad. This keeps credentials, ownership, scheduling, and troubleshooting isolated.

---

## 2. Why this component exists

Running generative-quality evaluation synchronously inside the agent request path would increase latency, cost, and coupling. The evaluator moves that responsibility out of the online interaction path.

The architecture is:

```text
Online interaction
        |
        v
Agent Framework
        |
        +--------------------> Langfuse v3
                                  |
                                  | traces / sessions
                                  v
                         D+1 Evaluator (CronJob)
                                  |
               +------------------+------------------+
               |                  |                  |
               v                  v                  v
          OCI GenAI             ADB                GCS
        LLM-as-a-Judge     state/results       official .TXT.GZ
               |
               +-----------------------> Langfuse
                                      audit / optional scores
```

This separation provides:

- no evaluator latency in the online interaction;
- replay and auditability;
- configurable sampling per agent;
- isolation of evaluation failures from the online agent;
- support for dedicated OCI GenAI models;
- an official artifact for the TIM analytics layer;
- no code coupling to specific agent names, buckets, or credentials.

---

## 3. Core concepts

### 3.1. Trace

A **trace** represents an interaction evaluated individually. The evaluator calculates these trace-level metrics:

- `judgeScore` — overall response quality;
- `accuracyScore` — factual/contextual accuracy;
- `alucinationScore` — degree of unsupported claims;
- `promptLength` — raw character count of `trace.input`.

### 3.2. Session

A **session** represents the complete conversation. A session normally contains multiple traces.

Session-level metrics are calculated once and replicated to every trace row in that session:

- `inferredCsiScore`;
- `resolution`;
- `conversationPrecision`;
- `loop`.

### 3.3. Session-based sampling

Sampling is performed on **sessions**, never on isolated traces.

The deterministic seed is:

```text
agentId:reference_date
```

Example:

```text
billing:2026-08-13
```

Two executions for the same agent, date, and percentage therefore select the same sessions.

Once a session is selected, **all traces in that session** are retained.

### 3.4. BRT time window

`run-agents` calculates the reference date using:

```text
D - daysBack
```

with explicit timezone:

```text
America/Sao_Paulo
```

With `daysBack: 1`, a run on August 14 evaluates the full August 13 window, from 00:00:00 through 23:59:59.999999 BRT.

### 3.5. CONTEXT_TURN_LIMIT

`CONTEXT_TURN_LIMIT` controls how many previous conversation turns are included in trace-level judge context.

Default:

```env
CONTEXT_TURN_LIMIT=6
```

The variable exists because a response should not be judged outside the context of its conversation. Increasing the value provides more context, but also increases tokens, cost, and judge latency.

### 3.6. Loop

The `loop` metric is not simple text repetition. It represents the maximum consecutive **`metadata.intent` streak** in a session.

Example:

```text
billing
billing
billing
payment
payment
```

Result:

```text
loop = 3
```

---

## 4. End-to-end execution flow

The production flow is executed with:

```bash
python -m evaluator.cli run-agents
```

or through the installed entrypoint:

```bash
af-evaluator run-agents
```

Steps:

```text
1. Load agents.yaml
2. Compute D-daysBack in BRT
3. Resolve Langfuse credentials for the agent
4. Read ALL native sessions in the period, paginated
5. Read ALL traces in the period, paginated
6. Read observations in bulk/paginated mode
7. Recover the real session_id from metadata when a legacy trace has no native sessionId
8. Filter agent traces/aliases
9. Deterministically sample logical sessions
10. Keep all traces belonging to sampled sessions
11. Persist items in ADB
12. Run trace-level judge
13. Run session-level judge
14. Calculate local loop metric
15. Generate .TXT.GZ
16. Upload to GCS
17. Delete local file after successful upload
18. Update judge-job-{agentId} audit trace in Langfuse
```

### 4.1. Why collection avoids N+1

The implementation avoids the old pattern:

```text
1 trace
  -> GET trace detail
  -> GET observations
```

repeated once per trace.

Collection now uses bulk pagination:

```text
/api/public/sessions       page 1..N
/api/public/traces         page 1..N
/api/public/observations   page 1..N
```

Therefore 10,000 traces do not produce approximately 20,000 additional requests. Request count grows with the number of API pages instead.

The current `PAGE_SIZE` is 100.

### 4.2. Compatibility with traces without native `sessionId`

Some historical instrumentations store the session in trace/observation metadata without promoting it to Langfuse's native `trace.sessionId`. The evaluator recovers the real session using this precedence:

```text
trace.sessionId / trace.session_id
  -> trace.metadata.session_id / sessionId / conversation_key
  -> observation.metadata.session_id / sessionId / conversation_key
  -> business_context.session_key (last metadata fallback)
```

`trace.id` is **never** used as a `session_id`. This preserves session sampling and session-level metrics for historical traces without creating an artificial one-session-per-trace model. Collection statistics expose `native_sessions_read`, `metadata_sessions_recovered`, and `traces_without_session`.

---

## 5. Component architecture

```text
agent_framework_oci_evaluator/
├── evaluator/
│   ├── analytics/
│   │   └── vloop.py
│   ├── collectors/
│   │   ├── langfuse.py
│   │   ├── langfuse_credentials.py
│   │   ├── agent_framework.py
│   │   └── mock.py
│   ├── config/
│   │   ├── agents.py
│   │   └── settings.py
│   ├── core/
│   │   └── models.py
│   ├── identity/
│   │   └── resolver.py
│   ├── judges/
│   │   └── llm_judge.py
│   ├── llm/
│   │   ├── client.py
│   │   └── profile_resolver.py
│   ├── output/
│   │   └── legacy_exporter.py
│   ├── persistence/
│   │   ├── oracle_store.py
│   │   └── repository.py
│   ├── prompts/
│   │   └── loader.py
│   ├── publishers/
│   │   ├── gcs_export.py
│   │   ├── langfuse_job_audit.py
│   │   └── langfuse_scores.py
│   ├── cli.py
│   ├── engine.py
│   └── time_utils.py
├── configs/
│   ├── identity.yaml
│   ├── judge/
│   │   ├── agents.yaml
│   │   ├── storage.yaml
│   │   ├── trace_metrics.yaml
│   │   └── session_metrics.yaml
│   ├── langfuse/
│   │   └── langfuse_credentials.example.yaml
│   └── llm_profiles/
│       └── llm_profiles.yaml
├── deploy/
├── tests/
├── .env.example
├── Dockerfile
└── pyproject.toml
```

### 5.1. Collectors

Collectors transform external data sources into `ConversationRecord` objects.

Available sources:

```text
langfuse
agent_framework
mock
```

`langfuse` is the default production source.

### 5.2. EvaluationEngine

`evaluator/engine.py` orchestrates:

- run creation;
- collection;
- sampling;
- persistence;
- judging;
- export;
- upload;
- audit.

### 5.3. Persistence

Execution state and results are persisted in Oracle Database. The schema is created automatically by `init-db`.

Main tables:

```text
EVALUATION_RUN
EVALUATION_PROGRESS_EVENT
EVALUATION_ITEM
EVALUATION_RESULT
EVALUATION_METRIC
EVALUATION_FINDING
```

The table prefix is configured through:

```env
ADB_TABLE_PREFIX=AGENTFW
```

### 5.4. Publishers

- `gcs_export.py` — uploads the official artifact;
- `langfuse_job_audit.py` — writes the `judge-job-{agentId}` audit trace;
- `langfuse_scores.py` — optionally publishes trace scores back to Langfuse.

> **Note:** production job audit uses the agent's dynamic credentials. Optional score publishing (`PUBLISH_LANGFUSE_SCORES=true`) still uses global `LANGFUSE_HOST/PUBLIC_KEY/SECRET_KEY`; keep it disabled unless an appropriate global project is configured.

---

## 6. Agent configuration

Default file:

```text
configs/judge/agents.yaml
```

Example:

```yaml
agents:
  - agentId: billing
    enabled: true
    daysBack: 1
    percentage: 1.0
    langfuse_agent_aliases:
      - telecom_contas
    gcs_prefix: agnt_ai_contas
```

Fields:

| Field | Description |
|---|---|
| `agentId` | official logical agent identity |
| `enabled` | enables/disables the agent in that deployment |
| `daysBack` | number of days to subtract from the current BRT date |
| `percentage` | percentage of sessions evaluated, from 0.0 to 1.0 |
| `langfuse_agent_aliases` | technical agent names accepted in traces; when populated, it is an explicit filter regardless of the Langfuse credential source |
| `gcs_prefix` | agent prefix inside the bucket |

`agent_id` and `days_back` are also accepted for compatibility, but `agentId` / `daysBack` is the recommended form.

**`langfuse_agent_aliases` semantics:** when the list contains values, only traces whose agent identifier matches one of the aliases are eligible for sampling and evaluation. When the list is empty, the evaluator does not apply an agent-identity filter and treats the configured Langfuse project/endpoint as exclusive to that entry.

### 6.1. One evaluator per agent

For the recommended one-instance-per-agent deployment model, keep only one agent enabled in that deployment:

```yaml
agents:
  - agentId: billing
    enabled: true
    daysBack: 1
    percentage: 1.0
    langfuse_agent_aliases:
      - telecom_contas
      - billing_agent
    gcs_prefix: agnt_ai_contas
```

The same evaluator code can be reused by another repository/deployment simply by changing configuration files.

---

## 7. Dynamic Langfuse credentials

There are no hardcoded variables such as:

```text
LANGFUSE_BILLING_*
LANGFUSE_SALES_*
```

The evaluator reads one declarative definition:

```env
LANGFUSE_CREDENTIALS_CONFIG_PATH=configs/langfuse/langfuse_credentials.yaml
```

Create it from the example:

```bash
cp configs/langfuse/langfuse_credentials.example.yaml \
   configs/langfuse/langfuse_credentials.yaml
```

Example:

```yaml
langfuse:
  agents:
    - agentId: billing
      host: https://langfuse.example.internal
      publicKey: pk-...
      secretKey: sk-...

    - agentId: another-agent
      host: https://another-langfuse.internal
      publicKey: pk-...
      secretKey: sk-...
```

The resolver iterates through all entries. Adding a new agent requires no Python change.

### 7.1. Environment/file references

To avoid storing secrets in Git:

```yaml
langfuse:
  agents:
    - agentId: billing
      hostEnv: BILLING_LF_HOST
      publicKeyEnv: BILLING_LF_PUBLIC
      secretKeyFile: /var/secrets/langfuse/billing-secret-key
```

Supported fields:

```text
host / hostEnv / hostFile
publicKey / publicKeyEnv / publicKeyFile
secretKey / secretKeyEnv / secretKeyFile
```

Resolution order for each field:

```text
direct value -> declared environment variable -> declared file
```

### 7.2. Global credentials

These variables remain available only for the generic manual `run` command:

```env
LANGFUSE_HOST=http://localhost:3005
LANGFUSE_PUBLIC_KEY=...
LANGFUSE_SECRET_KEY=...
```

The production `run-agents` flow resolves credentials from the per-agent declarative file.

---

## 8. LLM configuration

Supported providers:

```text
mock
oci_openai
oci_sdk
openai_compatible
```

The recommended provider for dedicated OCI models is:

```env
LLM_PROVIDER=oci_sdk
```

### 8.1. OCI SDK

Local example using `~/.oci/config`:

```env
LLM_PROVIDER=oci_sdk
OCI_AUTH_MODE=config_file
OCI_CONFIG_FILE=~/.oci/config
OCI_PROFILE=DEFAULT
OCI_REGION=sa-saopaulo-1
OCI_COMPARTMENT_ID=ocid1.compartment...
OCI_GENAI_BASE_URL=https://inference.generativeai.sa-saopaulo-1.oci.oraclecloud.com
OCI_GENAI_MODEL=meta.llama-3.3-70b-instruct
```

Authentication modes supported by the code:

```text
config_file
instance_principal
resource_principal
oke_workload_identity
workload_identity
```

### 8.2. Dedicated endpoint

For a dedicated endpoint:

```env
OCI_GENAI_BASE_URL=https://<private-endpoint>.inference.generativeai.<region>.oci.oraclecloud.com
OCI_GENAI_ENDPOINT_ID=ocid1.generativeaiendpoint...
```

The dedicated endpoint OCID may also be provided in `OCI_GENAI_MODEL`.

The provider selects `DedicatedServingMode` when an endpoint OCID is available; otherwise it uses `OnDemandServingMode`.

### 8.3. Judge profiles

File:

```text
configs/llm_profiles/llm_profiles.yaml
```

Example:

```yaml
profiles:
  judge:
    temperature: 0
    max_tokens: 900
```

Selected through:

```env
LLM_PROFILE=judge
```

---

## 9. Prompts and metrics

### 9.1. Trace metrics

File:

```text
configs/judge/trace_metrics.yaml
```

The judge must return JSON such as:

```json
{
  "judgeScore": 0.0,
  "accuracyScore": 0.0,
  "alucinationScore": 0.0,
  "rationale": "..."
}
```

If parsing fails:

```text
1st attempt
   -> parse fails
2nd attempt with retry hint
   -> parse fails again
trace-level metrics = None
   -> exported as "0.00"
```

### 9.2. Session metrics

File:

```text
configs/judge/session_metrics.yaml
```

Expected result:

```json
{
  "inferredCsiScore": 0.5,
  "resolution": 1,
  "conversationPrecision": 1,
  "rationale": "..."
}
```

Session-level values are replicated to every trace row belonging to that session.

---

## 10. Identity normalization

File:

```text
configs/identity.yaml
```

The identity layer converts channel/domain-specific identifiers into canonical keys such as:

```text
customer_key
contract_key
interaction_key
account_key
resource_key
session_key
```

This prevents `cpf`, `msisdn`, `invoice_id`, `order_id`, `ura_call_id`, and similar names from being spread throughout evaluator code.

Source ordering works as fallback priority.

Example:

```yaml
session_key:
  sources:
    - business_context.session_key
    - session_key
    - conversation_key
    - session_id
```

---

## 11. GCS configuration

Buckets live outside the code in:

```text
configs/judge/storage.yaml
```

Example:

```yaml
storage:
  buckets:
    dev: tim-bigdata-dev-ca1f-raw-standard-agnt-ai-atendimento
    fqa: tim-bigdata-fqa-60ce-raw-standard-agnt-ai-atendimento
    prd: tim-bigdata-prod-e305-raw-standard-agnt-ai-atendimento
```

Selection:

```env
APP_ENV=dev
STORAGE_CONFIG_PATH=configs/judge/storage.yaml
```

Operational override:

```env
JUDGE_GCS_BUCKET=my-bucket
```

Enable the production flow with:

```env
ENABLE_GCS_UPLOAD=true
GOOGLE_APPLICATION_CREDENTIALS=/path/GCP_ACCESS_KEY.json
```

`run-agents` treats GCS as mandatory. If `ENABLE_GCS_UPLOAD=false` or upload fails, the agent run finishes as `FAILED`.

---

## 12. Output file

Pattern:

```text
AGENTE_{agentId}_LLM_JUDGE_{YYYYMMDD}_{HHMMSS}.TXT.GZ
```

Example:

```text
AGENTE_billing_LLM_JUDGE_20260814_020015.TXT.GZ
```

Header:

```text
"judgeScore"|;"accuracyScore"|;"alucinationScore"|;"promptLength"|;"loop"|;"inferredCsiScore"|;"resolution"|;"conversationPrecision"|;"uraCallId"|;"channelId"|;"sessionId"|;"messageId"
```

Final line:

```text
"TOTAL"|;"<n_traces>"
```

After a successful GCS upload, the local file is deleted.

---

## 13. Local installation

### 13.1. Prerequisites

- Python 3.11+;
- access to Oracle Database/Autonomous Database;
- Oracle Wallet when required;
- access to Langfuse v3;
- OCI Generative AI access for real judge execution;
- GCP credentials for production upload.

### 13.2. Create the virtual environment

Linux/WSL/macOS:

```bash
python -m venv .venv
source .venv/bin/activate
```

Windows PowerShell:

```powershell
python -m venv .venv
.\.venv\Scripts\Activate.ps1
```

### 13.3. Install

Recommended development mode:

```bash
python -m pip install --upgrade pip setuptools wheel
pip install -e .
```

Validate:

```bash
pip check
python -c "import oci, httpx, oracledb; import google.cloud.storage; print('Dependencies OK')"
```

---

## 14. Local configuration step by step

### Step 1 — `.env`

```bash
cp .env.example .env
```

Edit the real values.

Typical minimum configuration:

```env
APP_ENV=dev

ADB_USER=admin
ADB_PASSWORD=...
ADB_DSN=...
ADB_WALLET_LOCATION=/path/to/wallet
ADB_WALLET_PASSWORD=...
ADB_TABLE_PREFIX=AGENTFW

ENABLE_LANGFUSE=true
LANGFUSE_CREDENTIALS_CONFIG_PATH=configs/langfuse/langfuse_credentials.yaml

LLM_PROVIDER=oci_sdk
LLM_PROFILE=judge
OCI_AUTH_MODE=config_file
OCI_CONFIG_FILE=~/.oci/config
OCI_PROFILE=DEFAULT
OCI_REGION=sa-saopaulo-1
OCI_COMPARTMENT_ID=...
OCI_GENAI_BASE_URL=https://...
OCI_GENAI_ENDPOINT_ID=ocid1.generativeaiendpoint...

CONTEXT_TURN_LIMIT=6
BATCH_SIZE=50
MAX_ATTEMPTS=3

ENABLE_GCS_UPLOAD=true
STORAGE_CONFIG_PATH=configs/judge/storage.yaml
GOOGLE_APPLICATION_CREDENTIALS=/path/GCP_ACCESS_KEY.json
```

### Step 2 — Langfuse

```bash
cp configs/langfuse/langfuse_credentials.example.yaml \
   configs/langfuse/langfuse_credentials.yaml
```

Declare the agent that will be executed.

### Step 3 — agents.yaml

Adjust:

```text
configs/judge/agents.yaml
```

For a dedicated instance, keep only that squad's agent enabled.

### Step 4 — validate configuration

```bash
python -m evaluator.cli show-config
```

### Step 5 — initialize the database

```bash
python -m evaluator.cli init-db
```

> `reset-db` drops and recreates the evaluator schema. Use it only in a controlled environment.

---

## 15. CLI

The actual help for the current version is:

```bash
python -m evaluator.cli --help
```

Available commands:

```text
init-db
reset-db
show-config
run
run-agents
progress
runs
```

### 15.1. `run` — manual/diagnostic execution

Syntax:

```bash
python -m evaluator.cli run PERIOD_START PERIOD_END [OPTIONS]
```

Example:

```bash
python -m evaluator.cli run \
  2026-08-12T00:00:00 \
  2026-08-12T23:59:59 \
  --source langfuse \
  --limit 10
```

Options:

```text
--source TEXT                default: langfuse
--limit INTEGER
--show-progress / --no-show-progress
```

**Important:** `run` is a generic manual path. It collects, evaluates, and persists results, but does not execute the full production per-agent contract (configured sampling, GCS export, and job audit). Use `run-agents` for certification/production behavior.

### 15.2. `run-agents` — production flow

```bash
python -m evaluator.cli run-agents
```

One agent:

```bash
python -m evaluator.cli run-agents \
  --agent-id billing \
  --source langfuse
```

Limited test:

```bash
python -m evaluator.cli run-agents \
  --agent-id billing \
  --limit 10
```

Options:

```text
--source TEXT       default: langfuse
--agent-id TEXT
--limit INTEGER
```

`run-agents` uses the agent's `daysBack`, `percentage`, credentials, and `gcs_prefix`.

### 15.3. List runs

```bash
python -m evaluator.cli runs
```

or:

```bash
python -m evaluator.cli runs --limit 50
```

### 15.4. Inspect run progress

```bash
python -m evaluator.cli progress <RUN_ID>
```

With more events:

```bash
python -m evaluator.cli progress <RUN_ID> --events 100
```

Typical production events:

```text
LANGFUSE_JOB_AUDIT
RUN_CREATED
COLLECTING
COLLECTED
SAMPLED
ITEMS_INSERTED
BATCH_STARTED
ITEM_COMPLETED
SESSION_JUDGE_COMPLETED
COMPLETED
EXPORTED
GCS_UPLOADED
LOCAL_FILE_DELETED
LANGFUSE_JOB_AUDIT
```

---

## 16. Tests

Install `pytest` if needed:

```bash
pip install pytest
```

### 16.1. All tests

```bash
python -m pytest -v
```

### 16.2. TIM specification tests

```bash
python -m pytest tests/test_tim_spec.py -v
```

### 16.3. Extended compliance tests

```bash
python -m pytest tests/test_full_tim_compliance.py -v
```

### 16.4. Hardcode regression tests

```bash
python -m pytest tests/test_no_product_hardcodes.py -v
```

### 16.5. Recommended release suite

```bash
python -m pytest \
  tests/test_tim_spec.py \
  tests/test_full_tim_compliance.py \
  tests/test_no_product_hardcodes.py \
  -v
```

Validate syntax/imports:

```bash
python -m compileall evaluator
```

---

## 17. Recommended local functional test

### 17.1. Mock smoke test

Configure:

```env
LLM_PROVIDER=mock
```

Then run a small period using the mock source when desired:

```bash
python -m evaluator.cli run \
  2026-08-12T00:00:00 \
  2026-08-12T23:59:59 \
  --source mock
```

### 17.2. Small Langfuse test

```bash
python -m evaluator.cli run-agents \
  --agent-id billing \
  --limit 5
```

Then:

```bash
python -m evaluator.cli runs --limit 5
python -m evaluator.cli progress <RUN_ID> --events 100
```

Confirm:

- `COLLECTED` > 0;
- `SAMPLED` lists selected sessions;
- items finish without `ITEM_FAILED`;
- `EXPORTED` appears;
- `GCS_UPLOADED` contains a `gs://...` URI;
- `LOCAL_FILE_DELETED` appears;
- `completed` audit was confirmed.

---

## 18. Docker

Local build:

```bash
docker build -t agent-framework-evaluator:local .
```

The production runtime is batch-only. The container does not need to expose an HTTP port.

In Kubernetes, the `CronJob` runs:

```text
python -m evaluator.cli run-agents --source langfuse
```

---

## 19. Kubernetes / OKE

The **evaluator itself remains CronJob-only**, as required by the specification: it does not open an HTTP/gRPC port and evaluation processing continues to run exclusively in `evaluator-batch`.

The web interface is provided by a **separate read-only component**, `evaluator-monitor`. It does not execute evaluations, mutate runs, or change the evaluator batch contract. It only reads run/progress/result tables from Autonomous Database.

```text
Evaluator batch (evaluator-batch CronJob)
        │
        ├── writes EVALUATION_RUN
        ├── writes EVALUATION_PROGRESS_EVENT
        └── writes EVALUATION_RESULT
                    │
                    ▼
             Autonomous Database
                    ▲
                    │ read-only
        Evaluator Monitor (HTTP Deployment)
                    │
                    └── /ui + monitoring APIs
```

Kustomize resources:

```text
Namespace
ServiceAccount
ConfigMap
CronJob evaluator-batch
Deployment evaluator-monitor
Service evaluator-monitor (ClusterIP)
```

No Ingress is created by default, so the UI is not automatically exposed outside the cluster.

Current CronJob:

```yaml
schedule: "0 2 * * *"
timeZone: "America/Sao_Paulo"
suspend: false
concurrencyPolicy: Forbid
```

### 19.1. Prepare deployment files

```bash
cp deploy/env/evaluator.env.example deploy/env/evaluator.env
cp deploy/langfuse/langfuse_credentials.example.yaml \
   deploy/langfuse/langfuse_credentials.yaml
```

Place the Oracle wallet in:

```text
deploy/wallet/
```

GCP credential:

```text
deploy/gcp/GCP_ACCESS_KEY.json
```

### 19.2. Create secrets

DEV:

```bash
./deploy/scripts/create-secrets.sh dev
```

HML:

```bash
./deploy/scripts/create-secrets.sh hml
```

PROD:

```bash
./deploy/scripts/create-secrets.sh prod
```

Created/updated secrets:

```text
evaluator-secrets
evaluator-wallet
evaluator-langfuse
evaluator-gcp        (when the GCP file exists)
```

### 19.3. Build and push

```bash
export REGISTRY=tim.azurecr.io
export IMAGE_TAG=<version>
./deploy/scripts/build-push.sh prod
```

### 19.4. Deploy

```bash
export REGISTRY=tim.azurecr.io
export IMAGE_TAG=<version>
./deploy/scripts/deploy.sh prod
```

By default the script also runs the database initialization job.

To disable it:

```bash
export RUN_INIT_DB=false
```

### 19.5. Status

```bash
./deploy/scripts/status.sh prod
```

or directly:

```bash
kubectl -n agent-evaluator-prod get cronjob,jobs,pods
```

### 19.6. Immediate execution outside the schedule

```bash
./deploy/scripts/run-now.sh prod
```

The script creates a Job from the CronJob.

Follow logs:

```bash
kubectl -n agent-evaluator-prod get jobs,pods
kubectl -n agent-evaluator-prod logs -f job/<job-name>
```


### 19.7. Web monitoring interface

The UI is served by the independent `evaluator-monitor` component:

```text
GET /ui
GET /health
GET /api/runs
GET /api/runs/{run_id}
GET /api/runs/{run_id}/results
```

To access a Kubernetes deployment locally without creating an Ingress:

```bash
kubectl -n agent-evaluator-prod port-forward svc/evaluator-monitor 8080:8080
```

Then open:

```text
http://localhost:8080/ui
```

The page shows recent runs, `RUN_ID`, agent, status, progress, events, and results. The browser auto-refreshes every 5 seconds.

To run the monitor locally:

```bash
python -m uvicorn evaluator_monitor.app:app --host 0.0.0.0 --port 8080
```

The monitor uses the same `ADB_*`/wallet settings as the evaluator. Functionally it is **read-only**: there is no endpoint to start, cancel, delete, or modify runs.

> Security recommendation: keep the Service as `ClusterIP`. If external exposure is required later, enforce authentication/authorization through the corporate Ingress/API Gateway; do not expose the monitor directly to the Internet.

---

## 20. How to release a new version

Recommended flow:

### 1. Update code/configuration

Never commit real credentials.

### 2. Run tests

```bash
python -m pytest -v
python -m compileall evaluator
```

### 3. Test locally with a small limit

```bash
python -m evaluator.cli run-agents \
  --agent-id <agentId> \
  --limit 5
```

### 4. Validate the run

```bash
python -m evaluator.cli runs --limit 10
python -m evaluator.cli progress <RUN_ID> --events 100
```

### 5. Confirm GCS

The event stream must include:

```text
GCS_UPLOADED
```

with a URI such as:

```text
gs://<bucket>/<prefix>/AGENTE_<agent>_LLM_JUDGE_<timestamp>.TXT.GZ
```

### 6. Update environment secrets/configuration

```bash
./deploy/scripts/create-secrets.sh <dev|hml|prod>
```

### 7. Build/push a new tag

```bash
export REGISTRY=<registry>
export IMAGE_TAG=<new-version>
./deploy/scripts/build-push.sh <environment>
```

### 8. Deploy

```bash
./deploy/scripts/deploy.sh <environment>
```

### 9. Trigger a manual certification run

```bash
./deploy/scripts/run-now.sh <environment>
```

### 10. Validate logs and artifacts

```bash
kubectl -n agent-evaluator-<environment> logs -f job/<job>
```

Also confirm the GCS object and the `judge-job-{agentId}` audit in Langfuse.

---

## 21. Troubleshooting

### `Collected 0 traces`

Check, in this order:

1. processed date (`daysBack` uses BRT);
2. Langfuse credentials for the correct `agentId`;
3. Langfuse project `host`;
4. `langfuse_agent_aliases`;
5. whether sessions/traces exist in the period;
6. whether the logical `agentId` differs from `metadata.agent_id` in traces.

Example:

```yaml
- agentId: billing
  langfuse_agent_aliases:
    - telecom_contas
    - billing_agent
```

### `No Langfuse credentials configured for agent ...`

The agent exists in `agents.yaml` but not in the file referenced by:

```env
LANGFUSE_CREDENTIALS_CONFIG_PATH
```

Add an entry with the same `agentId`.

### `Langfuse credentials definition file not found`

Create/mount:

```text
configs/langfuse/langfuse_credentials.yaml
```

or configure another path.

### `TIM specification requires GCS upload...`

`run-agents` was executed with:

```env
ENABLE_GCS_UPLOAD=false
```

Set it to `true` and configure GCP credentials.

### GCS upload failure

Check:

```bash
echo "$GOOGLE_APPLICATION_CREDENTIALS"
ls -l "$GOOGLE_APPLICATION_CREDENTIALS"
```

Also validate the bucket resolved through `APP_ENV`/`storage.yaml`.

### OCI SDK dedicated endpoint failure

Check:

```env
LLM_PROVIDER=oci_sdk
OCI_GENAI_BASE_URL=https://<native-endpoint-without-openai-v1>
OCI_GENAI_ENDPOINT_ID=ocid1.generativeaiendpoint...
OCI_COMPARTMENT_ID=...
```

Do not use `/openai/v1` in the native `oci_sdk` endpoint.

### Database schema missing

```bash
python -m evaluator.cli init-db
```

### View progress without a UI

The service exposes no HTTP endpoint. Use:

```bash
python -m evaluator.cli runs
python -m evaluator.cli progress <RUN_ID> --events 100
```

---

## 22. Main environment variables

| Variable | Purpose |
|---|---|
| `APP_ENV` | environment used to resolve storage |
| `ADB_*` | Oracle connection and wallet |
| `ENABLE_LANGFUSE` | enables Langfuse collection |
| `LANGFUSE_CREDENTIALS_CONFIG_PATH` | declarative per-agent YAML |
| `LANGFUSE_*` | global fallback only for manual `run` |
| `PUBLISH_LANGFUSE_SCORES` | publishes trace scores back to Langfuse |
| `LLM_PROVIDER` | `mock`, `oci_openai`, `oci_sdk`, `openai_compatible` |
| `LLM_PROFILE` | judge profile |
| `OCI_AUTH_MODE` | OCI SDK authentication mode |
| `OCI_GENAI_BASE_URL` | OCI GenAI endpoint |
| `OCI_GENAI_MODEL` | model ID/name or endpoint-OCID compatibility |
| `OCI_GENAI_ENDPOINT_ID` | dedicated endpoint OCID |
| `OCI_COMPARTMENT_ID` | OCI compartment |
| `AGENTS_CONFIG_PATH` | agents file |
| `TRACE_PROMPT_PATH` | trace-level prompt |
| `SESSION_PROMPT_PATH` | session-level prompt |
| `BATCH_SIZE` | internal processing batch size |
| `MAX_ATTEMPTS` | configurable attempt limit |
| `CONTEXT_TURN_LIMIT` | last N turns passed to trace judge |
| `ENABLE_GCS_UPLOAD` | requires upload in production flow |
| `STORAGE_CONFIG_PATH` | bucket mapping by environment |
| `JUDGE_GCS_BUCKET` | bucket override |
| `GOOGLE_APPLICATION_CREDENTIALS` | GCP credentials |
| `OUTPUT_DIR` | temporary export directory |

---

## 23. Security and operations

- do not commit the real `.env`;
- do not commit the real `langfuse_credentials.yaml`;
- do not commit GCP service account JSON;
- do not commit Oracle Wallet files;
- in OKE, prefer `oke_workload_identity` where applicable;
- mount credentials as Kubernetes Secrets;
- keep `readOnlyRootFilesystem` enabled in the CronJob;
- keep `concurrencyPolicy: Forbid` to prevent two D+1 jobs from the same deployment running concurrently;
- never run `reset-db` in production without an explicit recovery plan.

---

## 24. Certification checklist

Before promoting a version:

- [ ] `pip check` passes;
- [ ] `python -m pytest -v` passes;
- [ ] `python -m compileall evaluator` passes;
- [ ] `show-config` points to the expected ADB/OCI/config files;
- [ ] `agents.yaml` contains only the expected agents for that deployment;
- [ ] Langfuse credentials exist for every enabled `agentId`;
- [ ] `run-agents --limit 5` finds traces;
- [ ] sampling operates on sessions, not individual traces;
- [ ] trace/session judges complete;
- [ ] output contains the 12-column header;
- [ ] filename matches `AGENTE_{agentId}_LLM_JUDGE_{YYYYMMDD}_{HHMMSS}.TXT.GZ`;
- [ ] `GCS_UPLOADED` event is confirmed;
- [ ] local file is deleted after upload;
- [ ] Langfuse `created/completed` audit is confirmed;
- [ ] CronJob has `suspend: false`;
- [ ] CronJob timezone is `America/Sao_Paulo`;
- [ ] no real credentials are present in the repository.

---

## 25. Quick mental model for new contributors

If you need to understand the project quickly, think of it as:

```text
Langfuse stores conversations
        ↓
Evaluator selects D+1 sessions
        ↓
OCI GenAI judges traces and sessions
        ↓
ADB stores execution state and results
        ↓
GCS receives the official artifact
        ↓
Langfuse stores the batch audit trace
```

Extension points are configuration-driven:

```text
Which agent?           -> configs/judge/agents.yaml
Which Langfuse project?-> langfuse_credentials.yaml
Which bucket?          -> configs/judge/storage.yaml
Which prompt?          -> trace_metrics.yaml / session_metrics.yaml
Which model?           -> .env + llm_profiles.yaml
Which identity rules?  -> configs/identity.yaml
```

The most important architectural rule is: **the evaluator code must remain generic; agent, environment, credential, and storage details belong in configuration.**