cristiano.hoshikawa/agent_framework_llm_as_a_judge

Fork 0

Files

Cristiano Hoshikawa b2a368d71a first commit

2026-09-09 16:50:00 -03:00

32 KiB

Raw Blame History

Agent Framework OCI Evaluator — LLM-as-a-Judge

Language: English | Português

1. Overview

The Agent Framework OCI Evaluator is the batch component responsible for asynchronously evaluating the quality of conversations produced by agents built on the Agent Framework OCI.

The service reads sessions and traces from Langfuse v3, deterministically samples sessions, calculates quality metrics with an LLM-as-a-Judge, persists state and results in Oracle Autonomous Database, generates the official .TXT.GZ artifact, publishes that artifact to Google Cloud Storage (GCS), and audits the evaluator job itself in Langfuse.

It is designed for D+1 execution as a Kubernetes CronJob, with no runtime HTTP/gRPC endpoint.

Recommended deployment model: one evaluator instance per agent/squad. The code is generic and supports multiple agents in configuration, but in production each deployment should normally enable only the agent owned by that squad. This keeps credentials, ownership, scheduling, and troubleshooting isolated.

2. Why this component exists

Running generative-quality evaluation synchronously inside the agent request path would increase latency, cost, and coupling. The evaluator moves that responsibility out of the online interaction path.

The architecture is:

Online interaction
        |
        v
Agent Framework
        |
        +--------------------> Langfuse v3
                                  |
                                  | traces / sessions
                                  v
                         D+1 Evaluator (CronJob)
                                  |
               +------------------+------------------+
               |                  |                  |
               v                  v                  v
          OCI GenAI             ADB                GCS
        LLM-as-a-Judge     state/results       official .TXT.GZ
               |
               +-----------------------> Langfuse
                                      audit / optional scores

This separation provides:

no evaluator latency in the online interaction;
replay and auditability;
configurable sampling per agent;
isolation of evaluation failures from the online agent;
support for dedicated OCI GenAI models;
an official artifact for the TIM analytics layer;
no code coupling to specific agent names, buckets, or credentials.

3. Core concepts

3.1. Trace

A trace represents an interaction evaluated individually. The evaluator calculates these trace-level metrics:

judgeScore — overall response quality;
accuracyScore — factual/contextual accuracy;
alucinationScore — degree of unsupported claims;
promptLength — raw character count of trace.input.

3.2. Session

A session represents the complete conversation. A session normally contains multiple traces.

Session-level metrics are calculated once and replicated to every trace row in that session:

inferredCsiScore;
resolution;
conversationPrecision;
loop.

3.3. Session-based sampling

Sampling is performed on sessions, never on isolated traces.

The deterministic seed is:

agentId:reference_date

Example:

billing:2026-08-13

Two executions for the same agent, date, and percentage therefore select the same sessions.

Once a session is selected, all traces in that session are retained.

3.4. BRT time window

run-agents calculates the reference date using:

D - daysBack

with explicit timezone:

America/Sao_Paulo

With daysBack: 1, a run on August 14 evaluates the full August 13 window, from 00:00:00 through 23:59:59.999999 BRT.

3.5. CONTEXT_TURN_LIMIT

CONTEXT_TURN_LIMIT controls how many previous conversation turns are included in trace-level judge context.

Default:

CONTEXT_TURN_LIMIT=6

The variable exists because a response should not be judged outside the context of its conversation. Increasing the value provides more context, but also increases tokens, cost, and judge latency.

3.6. Loop

The loop metric is not simple text repetition. It represents the maximum consecutive metadata.intent streak in a session.

Example:

billing
billing
billing
payment
payment

Result:

loop = 3

4. End-to-end execution flow

The production flow is executed with:

python -m evaluator.cli run-agents

or through the installed entrypoint:

af-evaluator run-agents

Steps:

1. Load agents.yaml
2. Compute D-daysBack in BRT
3. Resolve Langfuse credentials for the agent
4. Read ALL native sessions in the period, paginated
5. Read ALL traces in the period, paginated
6. Read observations in bulk/paginated mode
7. Recover the real session_id from metadata when a legacy trace has no native sessionId
8. Filter agent traces/aliases
9. Deterministically sample logical sessions
10. Keep all traces belonging to sampled sessions
11. Persist items in ADB
12. Run trace-level judge
13. Run session-level judge
14. Calculate local loop metric
15. Generate .TXT.GZ
16. Upload to GCS
17. Delete local file after successful upload
18. Update judge-job-{agentId} audit trace in Langfuse

4.1. Why collection avoids N+1

The implementation avoids the old pattern:

1 trace
  -> GET trace detail
  -> GET observations

repeated once per trace.

Collection now uses bulk pagination:

/api/public/sessions       page 1..N
/api/public/traces         page 1..N
/api/public/observations   page 1..N

Therefore 10,000 traces do not produce approximately 20,000 additional requests. Request count grows with the number of API pages instead.

The current PAGE_SIZE is 100.

4.2. Compatibility with traces without native `sessionId`

Some historical instrumentations store the session in trace/observation metadata without promoting it to Langfuse's native trace.sessionId. The evaluator recovers the real session using this precedence:

trace.sessionId / trace.session_id
  -> trace.metadata.session_id / sessionId / conversation_key
  -> observation.metadata.session_id / sessionId / conversation_key
  -> business_context.session_key (last metadata fallback)

trace.id is never used as a session_id. This preserves session sampling and session-level metrics for historical traces without creating an artificial one-session-per-trace model. Collection statistics expose native_sessions_read, metadata_sessions_recovered, and traces_without_session.

5. Component architecture

agent_framework_oci_evaluator/
├── evaluator/
│   ├── analytics/
│   │   └── vloop.py
│   ├── collectors/
│   │   ├── langfuse.py
│   │   ├── langfuse_credentials.py
│   │   ├── agent_framework.py
│   │   └── mock.py
│   ├── config/
│   │   ├── agents.py
│   │   └── settings.py
│   ├── core/
│   │   └── models.py
│   ├── identity/
│   │   └── resolver.py
│   ├── judges/
│   │   └── llm_judge.py
│   ├── llm/
│   │   ├── client.py
│   │   └── profile_resolver.py
│   ├── output/
│   │   └── legacy_exporter.py
│   ├── persistence/
│   │   ├── oracle_store.py
│   │   └── repository.py
│   ├── prompts/
│   │   └── loader.py
│   ├── publishers/
│   │   ├── gcs_export.py
│   │   ├── langfuse_job_audit.py
│   │   └── langfuse_scores.py
│   ├── cli.py
│   ├── engine.py
│   └── time_utils.py
├── configs/
│   ├── identity.yaml
│   ├── judge/
│   │   ├── agents.yaml
│   │   ├── storage.yaml
│   │   ├── trace_metrics.yaml
│   │   └── session_metrics.yaml
│   ├── langfuse/
│   │   └── langfuse_credentials.example.yaml
│   └── llm_profiles/
│       └── llm_profiles.yaml
├── deploy/
├── tests/
├── .env.example
├── Dockerfile
└── pyproject.toml

5.1. Collectors

Collectors transform external data sources into ConversationRecord objects.

Available sources:

langfuse
agent_framework
mock

langfuse is the default production source.

5.2. EvaluationEngine

evaluator/engine.py orchestrates:

run creation;
collection;
sampling;
persistence;
judging;
export;
upload;
audit.

5.3. Persistence

Execution state and results are persisted in Oracle Database. The schema is created automatically by init-db.

Main tables:

EVALUATION_RUN
EVALUATION_PROGRESS_EVENT
EVALUATION_ITEM
EVALUATION_RESULT
EVALUATION_METRIC
EVALUATION_FINDING

The table prefix is configured through:

ADB_TABLE_PREFIX=AGENTFW

5.4. Publishers

gcs_export.py — uploads the official artifact;
langfuse_job_audit.py — writes the judge-job-{agentId} audit trace;
langfuse_scores.py — optionally publishes trace scores back to Langfuse.

Note: production job audit uses the agent's dynamic credentials. Optional score publishing (PUBLISH_LANGFUSE_SCORES=true) still uses global LANGFUSE_HOST/PUBLIC_KEY/SECRET_KEY; keep it disabled unless an appropriate global project is configured.

6. Agent configuration

Default file:

configs/judge/agents.yaml

Example:

agents:
  - agentId: billing
    enabled: true
    daysBack: 1
    percentage: 1.0
    langfuse_agent_aliases:
      - telecom_contas
    gcs_prefix: agnt_ai_contas

Fields:

Field	Description
`agentId`	official logical agent identity
`enabled`	enables/disables the agent in that deployment
`daysBack`	number of days to subtract from the current BRT date
`percentage`	percentage of sessions evaluated, from 0.0 to 1.0
`langfuse_agent_aliases`	technical agent names accepted in traces; when populated, it is an explicit filter regardless of the Langfuse credential source
`gcs_prefix`	agent prefix inside the bucket

agent_id and days_back are also accepted for compatibility, but agentId / daysBack is the recommended form.

langfuse_agent_aliases semantics: when the list contains values, only traces whose agent identifier matches one of the aliases are eligible for sampling and evaluation. When the list is empty, the evaluator does not apply an agent-identity filter and treats the configured Langfuse project/endpoint as exclusive to that entry.

6.1. One evaluator per agent

For the recommended one-instance-per-agent deployment model, keep only one agent enabled in that deployment:

agents:
  - agentId: billing
    enabled: true
    daysBack: 1
    percentage: 1.0
    langfuse_agent_aliases:
      - telecom_contas
      - billing_agent
    gcs_prefix: agnt_ai_contas

The same evaluator code can be reused by another repository/deployment simply by changing configuration files.

7. Dynamic Langfuse credentials

There are no hardcoded variables such as:

LANGFUSE_BILLING_*
LANGFUSE_SALES_*

The evaluator reads one declarative definition:

LANGFUSE_CREDENTIALS_CONFIG_PATH=configs/langfuse/langfuse_credentials.yaml

Create it from the example:

cp configs/langfuse/langfuse_credentials.example.yaml \
   configs/langfuse/langfuse_credentials.yaml

Example:

langfuse:
  agents:
    - agentId: billing
      host: https://langfuse.example.internal
      publicKey: pk-...
      secretKey: sk-...

    - agentId: another-agent
      host: https://another-langfuse.internal
      publicKey: pk-...
      secretKey: sk-...

The resolver iterates through all entries. Adding a new agent requires no Python change.

7.1. Environment/file references

To avoid storing secrets in Git:

langfuse:
  agents:
    - agentId: billing
      hostEnv: BILLING_LF_HOST
      publicKeyEnv: BILLING_LF_PUBLIC
      secretKeyFile: /var/secrets/langfuse/billing-secret-key

Supported fields:

host / hostEnv / hostFile
publicKey / publicKeyEnv / publicKeyFile
secretKey / secretKeyEnv / secretKeyFile

Resolution order for each field:

direct value -> declared environment variable -> declared file

7.2. Global credentials

These variables remain available only for the generic manual run command:

LANGFUSE_HOST=http://localhost:3005
LANGFUSE_PUBLIC_KEY=...
LANGFUSE_SECRET_KEY=...

The production run-agents flow resolves credentials from the per-agent declarative file.

8. LLM configuration

Supported providers:

mock
oci_openai
oci_sdk
openai_compatible

The recommended provider for dedicated OCI models is:

LLM_PROVIDER=oci_sdk

8.1. OCI SDK

Local example using ~/.oci/config:

LLM_PROVIDER=oci_sdk
OCI_AUTH_MODE=config_file
OCI_CONFIG_FILE=~/.oci/config
OCI_PROFILE=DEFAULT
OCI_REGION=sa-saopaulo-1
OCI_COMPARTMENT_ID=ocid1.compartment...
OCI_GENAI_BASE_URL=https://inference.generativeai.sa-saopaulo-1.oci.oraclecloud.com
OCI_GENAI_MODEL=meta.llama-3.3-70b-instruct

Authentication modes supported by the code:

config_file
instance_principal
resource_principal
oke_workload_identity
workload_identity

8.2. Dedicated endpoint

For a dedicated endpoint:

OCI_GENAI_BASE_URL=https://<private-endpoint>.inference.generativeai.<region>.oci.oraclecloud.com
OCI_GENAI_ENDPOINT_ID=ocid1.generativeaiendpoint...

The dedicated endpoint OCID may also be provided in OCI_GENAI_MODEL.

The provider selects DedicatedServingMode when an endpoint OCID is available; otherwise it uses OnDemandServingMode.

8.3. Judge profiles

File:

configs/llm_profiles/llm_profiles.yaml

Example:

profiles:
  judge:
    temperature: 0
    max_tokens: 900

Selected through:

LLM_PROFILE=judge

9. Prompts and metrics

9.1. Trace metrics

File:

configs/judge/trace_metrics.yaml

The judge must return JSON such as:

{
  "judgeScore": 0.0,
  "accuracyScore": 0.0,
  "alucinationScore": 0.0,
  "rationale": "..."
}

If parsing fails:

1st attempt
   -> parse fails
2nd attempt with retry hint
   -> parse fails again
trace-level metrics = None
   -> exported as "0.00"

9.2. Session metrics

File:

configs/judge/session_metrics.yaml

Expected result:

{
  "inferredCsiScore": 0.5,
  "resolution": 1,
  "conversationPrecision": 1,
  "rationale": "..."
}

Session-level values are replicated to every trace row belonging to that session.

10. Identity normalization

File:

configs/identity.yaml

The identity layer converts channel/domain-specific identifiers into canonical keys such as:

customer_key
contract_key
interaction_key
account_key
resource_key
session_key

This prevents cpf, msisdn, invoice_id, order_id, ura_call_id, and similar names from being spread throughout evaluator code.

Source ordering works as fallback priority.

Example:

session_key:
  sources:
    - business_context.session_key
    - session_key
    - conversation_key
    - session_id

11. GCS configuration

Buckets live outside the code in:

configs/judge/storage.yaml

Example:

storage:
  buckets:
    dev: tim-bigdata-dev-ca1f-raw-standard-agnt-ai-atendimento
    fqa: tim-bigdata-fqa-60ce-raw-standard-agnt-ai-atendimento
    prd: tim-bigdata-prod-e305-raw-standard-agnt-ai-atendimento

Selection:

APP_ENV=dev
STORAGE_CONFIG_PATH=configs/judge/storage.yaml

Operational override:

JUDGE_GCS_BUCKET=my-bucket

Enable the production flow with:

ENABLE_GCS_UPLOAD=true
GOOGLE_APPLICATION_CREDENTIALS=/path/GCP_ACCESS_KEY.json

run-agents treats GCS as mandatory. If ENABLE_GCS_UPLOAD=false or upload fails, the agent run finishes as FAILED.

12. Output file

Pattern:

AGENTE_{agentId}_LLM_JUDGE_{YYYYMMDD}_{HHMMSS}.TXT.GZ

Example:

AGENTE_billing_LLM_JUDGE_20260814_020015.TXT.GZ

Header:

"judgeScore"|;"accuracyScore"|;"alucinationScore"|;"promptLength"|;"loop"|;"inferredCsiScore"|;"resolution"|;"conversationPrecision"|;"uraCallId"|;"channelId"|;"sessionId"|;"messageId"

Final line:

"TOTAL"|;"<n_traces>"

After a successful GCS upload, the local file is deleted.

13. Local installation

13.1. Prerequisites

Python 3.11+;
access to Oracle Database/Autonomous Database;
Oracle Wallet when required;
access to Langfuse v3;
OCI Generative AI access for real judge execution;
GCP credentials for production upload.

13.2. Create the virtual environment

Linux/WSL/macOS:

python -m venv .venv
source .venv/bin/activate

Windows PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1

13.3. Install

Recommended development mode:

python -m pip install --upgrade pip setuptools wheel
pip install -e .

Validate:

pip check
python -c "import oci, httpx, oracledb; import google.cloud.storage; print('Dependencies OK')"

14. Local configuration step by step

Step 1 — `.env`

cp .env.example .env

Edit the real values.

Typical minimum configuration:

APP_ENV=dev

ADB_USER=admin
ADB_PASSWORD=...
ADB_DSN=...
ADB_WALLET_LOCATION=/path/to/wallet
ADB_WALLET_PASSWORD=...
ADB_TABLE_PREFIX=AGENTFW

ENABLE_LANGFUSE=true
LANGFUSE_CREDENTIALS_CONFIG_PATH=configs/langfuse/langfuse_credentials.yaml

LLM_PROVIDER=oci_sdk
LLM_PROFILE=judge
OCI_AUTH_MODE=config_file
OCI_CONFIG_FILE=~/.oci/config
OCI_PROFILE=DEFAULT
OCI_REGION=sa-saopaulo-1
OCI_COMPARTMENT_ID=...
OCI_GENAI_BASE_URL=https://...
OCI_GENAI_ENDPOINT_ID=ocid1.generativeaiendpoint...

CONTEXT_TURN_LIMIT=6
BATCH_SIZE=50
MAX_ATTEMPTS=3

ENABLE_GCS_UPLOAD=true
STORAGE_CONFIG_PATH=configs/judge/storage.yaml
GOOGLE_APPLICATION_CREDENTIALS=/path/GCP_ACCESS_KEY.json

Step 2 — Langfuse

cp configs/langfuse/langfuse_credentials.example.yaml \
   configs/langfuse/langfuse_credentials.yaml

Declare the agent that will be executed.

Step 3 — agents.yaml

Adjust:

configs/judge/agents.yaml

For a dedicated instance, keep only that squad's agent enabled.

Step 4 — validate configuration

python -m evaluator.cli show-config

Step 5 — initialize the database

python -m evaluator.cli init-db

reset-db drops and recreates the evaluator schema. Use it only in a controlled environment.

15. CLI

The actual help for the current version is:

python -m evaluator.cli --help

Available commands:

init-db
reset-db
show-config
run
run-agents
progress
runs

15.1. `run` — manual/diagnostic execution

Syntax:

python -m evaluator.cli run PERIOD_START PERIOD_END [OPTIONS]

Example:

python -m evaluator.cli run \
  2026-08-12T00:00:00 \
  2026-08-12T23:59:59 \
  --source langfuse \
  --limit 10

Options:

--source TEXT                default: langfuse
--limit INTEGER
--show-progress / --no-show-progress

Important: run is a generic manual path. It collects, evaluates, and persists results, but does not execute the full production per-agent contract (configured sampling, GCS export, and job audit). Use run-agents for certification/production behavior.

15.2. `run-agents` — production flow

python -m evaluator.cli run-agents

One agent:

python -m evaluator.cli run-agents \
  --agent-id billing \
  --source langfuse

Limited test:

python -m evaluator.cli run-agents \
  --agent-id billing \
  --limit 10

Options:

--source TEXT       default: langfuse
--agent-id TEXT
--limit INTEGER

run-agents uses the agent's daysBack, percentage, credentials, and gcs_prefix.

15.3. List runs

python -m evaluator.cli runs

or:

python -m evaluator.cli runs --limit 50

15.4. Inspect run progress

python -m evaluator.cli progress <RUN_ID>

With more events:

python -m evaluator.cli progress <RUN_ID> --events 100

Typical production events:

LANGFUSE_JOB_AUDIT
RUN_CREATED
COLLECTING
COLLECTED
SAMPLED
ITEMS_INSERTED
BATCH_STARTED
ITEM_COMPLETED
SESSION_JUDGE_COMPLETED
COMPLETED
EXPORTED
GCS_UPLOADED
LOCAL_FILE_DELETED
LANGFUSE_JOB_AUDIT

16. Tests

Install pytest if needed:

pip install pytest

16.1. All tests

python -m pytest -v

16.2. TIM specification tests

python -m pytest tests/test_tim_spec.py -v

16.3. Extended compliance tests

python -m pytest tests/test_full_tim_compliance.py -v

16.4. Hardcode regression tests

python -m pytest tests/test_no_product_hardcodes.py -v

16.5. Recommended release suite

python -m pytest \
  tests/test_tim_spec.py \
  tests/test_full_tim_compliance.py \
  tests/test_no_product_hardcodes.py \
  -v

Validate syntax/imports:

python -m compileall evaluator

17. Recommended local functional test

17.1. Mock smoke test

Configure:

LLM_PROVIDER=mock

Then run a small period using the mock source when desired:

python -m evaluator.cli run \
  2026-08-12T00:00:00 \
  2026-08-12T23:59:59 \
  --source mock

17.2. Small Langfuse test

python -m evaluator.cli run-agents \
  --agent-id billing \
  --limit 5

Then:

python -m evaluator.cli runs --limit 5
python -m evaluator.cli progress <RUN_ID> --events 100

Confirm:

COLLECTED > 0;
SAMPLED lists selected sessions;
items finish without ITEM_FAILED;
EXPORTED appears;
GCS_UPLOADED contains a gs://... URI;
LOCAL_FILE_DELETED appears;
completed audit was confirmed.

18. Docker

Local build:

docker build -t agent-framework-evaluator:local .

The production runtime is batch-only. The container does not need to expose an HTTP port.

In Kubernetes, the CronJob runs:

python -m evaluator.cli run-agents --source langfuse

19. Kubernetes / OKE

The evaluator itself remains CronJob-only, as required by the specification: it does not open an HTTP/gRPC port and evaluation processing continues to run exclusively in evaluator-batch.

The web interface is provided by a separate read-only component, evaluator-monitor. It does not execute evaluations, mutate runs, or change the evaluator batch contract. It only reads run/progress/result tables from Autonomous Database.

Evaluator batch (evaluator-batch CronJob)
        │
        ├── writes EVALUATION_RUN
        ├── writes EVALUATION_PROGRESS_EVENT
        └── writes EVALUATION_RESULT
                    │
                    ▼
             Autonomous Database
                    ▲
                    │ read-only
        Evaluator Monitor (HTTP Deployment)
                    │
                    └── /ui + monitoring APIs

Kustomize resources:

Namespace
ServiceAccount
ConfigMap
CronJob evaluator-batch
Deployment evaluator-monitor
Service evaluator-monitor (ClusterIP)

No Ingress is created by default, so the UI is not automatically exposed outside the cluster.

Current CronJob:

schedule: "0 2 * * *"
timeZone: "America/Sao_Paulo"
suspend: false
concurrencyPolicy: Forbid

19.1. Prepare deployment files

cp deploy/env/evaluator.env.example deploy/env/evaluator.env
cp deploy/langfuse/langfuse_credentials.example.yaml \
   deploy/langfuse/langfuse_credentials.yaml

Place the Oracle wallet in:

deploy/wallet/

GCP credential:

deploy/gcp/GCP_ACCESS_KEY.json

19.2. Create secrets

DEV:

./deploy/scripts/create-secrets.sh dev

HML:

./deploy/scripts/create-secrets.sh hml

PROD:

./deploy/scripts/create-secrets.sh prod

Created/updated secrets:

evaluator-secrets
evaluator-wallet
evaluator-langfuse
evaluator-gcp        (when the GCP file exists)

19.3. Build and push

export REGISTRY=tim.azurecr.io
export IMAGE_TAG=<version>
./deploy/scripts/build-push.sh prod

19.4. Deploy

export REGISTRY=tim.azurecr.io
export IMAGE_TAG=<version>
./deploy/scripts/deploy.sh prod

By default the script also runs the database initialization job.

To disable it:

export RUN_INIT_DB=false

19.5. Status

./deploy/scripts/status.sh prod

or directly:

kubectl -n agent-evaluator-prod get cronjob,jobs,pods

19.6. Immediate execution outside the schedule

./deploy/scripts/run-now.sh prod

The script creates a Job from the CronJob.

Follow logs:

kubectl -n agent-evaluator-prod get jobs,pods
kubectl -n agent-evaluator-prod logs -f job/<job-name>

19.7. Web monitoring interface

The UI is served by the independent evaluator-monitor component:

GET /ui
GET /health
GET /api/runs
GET /api/runs/{run_id}
GET /api/runs/{run_id}/results

To access a Kubernetes deployment locally without creating an Ingress:

kubectl -n agent-evaluator-prod port-forward svc/evaluator-monitor 8080:8080

Then open:

http://localhost:8080/ui

The page shows recent runs, RUN_ID, agent, status, progress, events, and results. The browser auto-refreshes every 5 seconds.

To run the monitor locally:

python -m uvicorn evaluator_monitor.app:app --host 0.0.0.0 --port 8080

The monitor uses the same ADB_*/wallet settings as the evaluator. Functionally it is read-only: there is no endpoint to start, cancel, delete, or modify runs.

Security recommendation: keep the Service as ClusterIP. If external exposure is required later, enforce authentication/authorization through the corporate Ingress/API Gateway; do not expose the monitor directly to the Internet.

20. How to release a new version

Recommended flow:

1. Update code/configuration

Never commit real credentials.

2. Run tests

python -m pytest -v
python -m compileall evaluator

3. Test locally with a small limit

python -m evaluator.cli run-agents \
  --agent-id <agentId> \
  --limit 5

4. Validate the run

python -m evaluator.cli runs --limit 10
python -m evaluator.cli progress <RUN_ID> --events 100

5. Confirm GCS

The event stream must include:

GCS_UPLOADED

with a URI such as:

gs://<bucket>/<prefix>/AGENTE_<agent>_LLM_JUDGE_<timestamp>.TXT.GZ

6. Update environment secrets/configuration

./deploy/scripts/create-secrets.sh <dev|hml|prod>

7. Build/push a new tag

export REGISTRY=<registry>
export IMAGE_TAG=<new-version>
./deploy/scripts/build-push.sh <environment>

8. Deploy

./deploy/scripts/deploy.sh <environment>

9. Trigger a manual certification run

./deploy/scripts/run-now.sh <environment>

10. Validate logs and artifacts

kubectl -n agent-evaluator-<environment> logs -f job/<job>

Also confirm the GCS object and the judge-job-{agentId} audit in Langfuse.

21. Troubleshooting

`Collected 0 traces`

Check, in this order:

processed date (daysBack uses BRT);
Langfuse credentials for the correct agentId;
Langfuse project host;
langfuse_agent_aliases;
whether sessions/traces exist in the period;
whether the logical agentId differs from metadata.agent_id in traces.

Example:

- agentId: billing
  langfuse_agent_aliases:
    - telecom_contas
    - billing_agent

`No Langfuse credentials configured for agent ...`

The agent exists in agents.yaml but not in the file referenced by:

LANGFUSE_CREDENTIALS_CONFIG_PATH

Add an entry with the same agentId.

`Langfuse credentials definition file not found`

Create/mount:

configs/langfuse/langfuse_credentials.yaml

or configure another path.

`TIM specification requires GCS upload...`

run-agents was executed with:

ENABLE_GCS_UPLOAD=false

Set it to true and configure GCP credentials.

GCS upload failure

Check:

echo "$GOOGLE_APPLICATION_CREDENTIALS"
ls -l "$GOOGLE_APPLICATION_CREDENTIALS"

Also validate the bucket resolved through APP_ENV/storage.yaml.

OCI SDK dedicated endpoint failure

Check:

LLM_PROVIDER=oci_sdk
OCI_GENAI_BASE_URL=https://<native-endpoint-without-openai-v1>
OCI_GENAI_ENDPOINT_ID=ocid1.generativeaiendpoint...
OCI_COMPARTMENT_ID=...

Do not use /openai/v1 in the native oci_sdk endpoint.

Database schema missing

python -m evaluator.cli init-db

View progress without a UI

The service exposes no HTTP endpoint. Use:

python -m evaluator.cli runs
python -m evaluator.cli progress <RUN_ID> --events 100

22. Main environment variables

Variable	Purpose
`APP_ENV`	environment used to resolve storage
`ADB_*`	Oracle connection and wallet
`ENABLE_LANGFUSE`	enables Langfuse collection
`LANGFUSE_CREDENTIALS_CONFIG_PATH`	declarative per-agent YAML
`LANGFUSE_*`	global fallback only for manual `run`
`PUBLISH_LANGFUSE_SCORES`	publishes trace scores back to Langfuse
`LLM_PROVIDER`	`mock`, `oci_openai`, `oci_sdk`, `openai_compatible`
`LLM_PROFILE`	judge profile
`OCI_AUTH_MODE`	OCI SDK authentication mode
`OCI_GENAI_BASE_URL`	OCI GenAI endpoint
`OCI_GENAI_MODEL`	model ID/name or endpoint-OCID compatibility
`OCI_GENAI_ENDPOINT_ID`	dedicated endpoint OCID
`OCI_COMPARTMENT_ID`	OCI compartment
`AGENTS_CONFIG_PATH`	agents file
`TRACE_PROMPT_PATH`	trace-level prompt
`SESSION_PROMPT_PATH`	session-level prompt
`BATCH_SIZE`	internal processing batch size
`MAX_ATTEMPTS`	configurable attempt limit
`CONTEXT_TURN_LIMIT`	last N turns passed to trace judge
`ENABLE_GCS_UPLOAD`	requires upload in production flow
`STORAGE_CONFIG_PATH`	bucket mapping by environment
`JUDGE_GCS_BUCKET`	bucket override
`GOOGLE_APPLICATION_CREDENTIALS`	GCP credentials
`OUTPUT_DIR`	temporary export directory

23. Security and operations

do not commit the real .env;
do not commit the real langfuse_credentials.yaml;
do not commit GCP service account JSON;
do not commit Oracle Wallet files;
in OKE, prefer oke_workload_identity where applicable;
mount credentials as Kubernetes Secrets;
keep readOnlyRootFilesystem enabled in the CronJob;
keep concurrencyPolicy: Forbid to prevent two D+1 jobs from the same deployment running concurrently;
never run reset-db in production without an explicit recovery plan.

24. Certification checklist

Before promoting a version:

pip check passes;
python -m pytest -v passes;
python -m compileall evaluator passes;
show-config points to the expected ADB/OCI/config files;
agents.yaml contains only the expected agents for that deployment;
Langfuse credentials exist for every enabled agentId;
run-agents --limit 5 finds traces;
sampling operates on sessions, not individual traces;
trace/session judges complete;
output contains the 12-column header;
filename matches AGENTE_{agentId}_LLM_JUDGE_{YYYYMMDD}_{HHMMSS}.TXT.GZ;
GCS_UPLOADED event is confirmed;
local file is deleted after upload;
Langfuse created/completed audit is confirmed;
CronJob has suspend: false;
CronJob timezone is America/Sao_Paulo;
no real credentials are present in the repository.

25. Quick mental model for new contributors

If you need to understand the project quickly, think of it as:

Langfuse stores conversations
        ↓
Evaluator selects D+1 sessions
        ↓
OCI GenAI judges traces and sessions
        ↓
ADB stores execution state and results
        ↓
GCS receives the official artifact
        ↓
Langfuse stores the batch audit trace

Extension points are configuration-driven:

Which agent?           -> configs/judge/agents.yaml
Which Langfuse project?-> langfuse_credentials.yaml
Which bucket?          -> configs/judge/storage.yaml
Which prompt?          -> trace_metrics.yaml / session_metrics.yaml
Which model?           -> .env + llm_profiles.yaml
Which identity rules?  -> configs/identity.yaml

The most important architectural rule is: the evaluator code must remain generic; agent, environment, credential, and storage details belong in configuration.

32 KiB Raw Blame History

Agent Framework OCI Evaluator — LLM-as-a-Judge

1. Overview

2. Why this component exists

3. Core concepts

3.1. Trace

3.2. Session

3.3. Session-based sampling

3.4. BRT time window

3.5. CONTEXT_TURN_LIMIT

3.6. Loop

4. End-to-end execution flow

4.1. Why collection avoids N+1

4.2. Compatibility with traces without native sessionId

5. Component architecture

5.1. Collectors

5.2. EvaluationEngine

5.3. Persistence

5.4. Publishers

6. Agent configuration

6.1. One evaluator per agent

7. Dynamic Langfuse credentials

7.1. Environment/file references

7.2. Global credentials

8. LLM configuration

8.1. OCI SDK

8.2. Dedicated endpoint

8.3. Judge profiles

9. Prompts and metrics

9.1. Trace metrics

9.2. Session metrics

10. Identity normalization

11. GCS configuration

12. Output file

13. Local installation

13.1. Prerequisites

13.2. Create the virtual environment

13.3. Install

14. Local configuration step by step

Step 1 — .env

Step 2 — Langfuse

Step 3 — agents.yaml

Step 4 — validate configuration

Step 5 — initialize the database

15. CLI

15.1. run — manual/diagnostic execution

15.2. run-agents — production flow

15.3. List runs

15.4. Inspect run progress

16. Tests

16.1. All tests

16.2. TIM specification tests

16.3. Extended compliance tests

16.4. Hardcode regression tests

16.5. Recommended release suite

17. Recommended local functional test

17.1. Mock smoke test

17.2. Small Langfuse test

18. Docker

19. Kubernetes / OKE

19.1. Prepare deployment files

19.2. Create secrets

19.3. Build and push

19.4. Deploy

19.5. Status

19.6. Immediate execution outside the schedule

19.7. Web monitoring interface

20. How to release a new version

1. Update code/configuration

2. Run tests

3. Test locally with a small limit

4. Validate the run

5. Confirm GCS

6. Update environment secrets/configuration

7. Build/push a new tag

8. Deploy

9. Trigger a manual certification run

10. Validate logs and artifacts

21. Troubleshooting

Collected 0 traces

32 KiB

Raw Blame History

4.2. Compatibility with traces without native `sessionId`

Step 1 — `.env`

15.1. `run` — manual/diagnostic execution

15.2. `run-agents` — production flow

`Collected 0 traces`

`No Langfuse credentials configured for agent ...`

`Langfuse credentials definition file not found`

`TIM specification requires GCS upload...`