cristiano.hoshikawa/agent_platform_oci
1
2
Fork 0
mirror of https://github.com/hoshikawa2/agent_platform_oci.git synced 2026-07-09 22:04:21 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
239203ee2a6070cb71f12d9b78289f0452f5e5de
agent_platform_oci/evals/offline/README.en-US.md
hoshikawa2 239203ee2a First commit
2026-06-19 22:17:09 -03:00

1547 lines
28 KiB
Markdown
Raw Blame History

# agent_framework_evaluator
## 1. What is the `agent_framework_evaluator`?
The `agent_framework_evaluator` is a complementary service to the `agent_framework_oci` created to evaluate real conversations conducted by the framework's agents.
It collects conversations from a source, usually Langfuse, reconstructs the context of the interaction, runs a Judge LLM, writes the results to an Oracle/ADB database, generates legacy files in TXT.GZ format, and optionally publishes scores back to Langfuse.
In simple terms:
```text
agent_framework_oci gera conversas e telemetria
↓
Langfuse armazena traces, spans, generations, metadata e usage
↓
agent_framework_evaluator coleta essas conversas
↓
LLM Judge avalia qualidade, precisão, alucinação, resolução e CSI
↓
Oracle/ADB persiste runs, itens, resultados, achados e progresso
↓
Exporter gera arquivo legado AGENTE_<agent>_LLM_JUDGE_YYYYMMDD.TXT.GZ
```
The evaluator does not replace the guardrails, online judges, or telemetry of `agent_framework_oci`. It acts as an offline/batch layer for evaluation, auditing, and export.
---
## 2. Purpose of the solution
The purpose of the evaluator is to allow conversations that have already taken place to be analyzed later using standardized criteria.
It mainly serves these scenarios:
- daily evaluation of conversations by agent;
- generation of legacy evaluation files;
- auditing the quality of responses;
- identification of hallucination, low accuracy, low resolution or poor customer experience;
- comparison between agents such as `telecom_contas`, `retail_orders` and `financeiro_agent`;
- optional publication of scores on Langfuse;
- persistence of evaluation history in Oracle/ADB;
- progress tracking via API or CLI.
---
## 3. How it integrates with `agent_framework_oci`
`agent_framework_oci` is the main runtime for agents. It executes the conversational flow with LangGraph, supervisor, guardrails, judges, MCP tools, memory, RAG, and telemetry.
During execution, the framework publishes traces to Langfuse containing:
- `trace_id`;
- `session_id`;
- `message_id`;
- `agent_id`;
- `channel`;
- canonical `business_context`;
- IC/NOC/GRL events;
- LangGraph spans;
- guardrail spans;
- judge spans;
- LLM generations;
- model usage, when available;
- `prompt_tokens`, `completion_tokens` and `total_tokens`, when returned by the provider;
- `input_size`, when emitted by the framework spans.
The evaluator uses this telemetry as a data source.
The main integration happens like this:
```text
agent_framework_oci
├── Executes agents
├── Resolves identity via identity.yaml
├── Creates canonical BusinessContext
├── Calls MCP/RAG/LLM
├── Emits Langfuse telemetry
└── Writes usage/model/tokens when available
agent_framework_evaluator
├── Reads traces in Langfuse
├── Applies identity.yaml to normalize identity
├── Rebuilds ConversationRecord
├── Executes LLM Judge offline
├── Writes results to Oracle/ADB
├── Exports legacy TXT.GZ
└── Optionally publish scores on Langfuse
```
---
## 4. General architecture
```text
+------------------------+
| agent_framework_oci |
|------------------------|
| LangGraph |
| Supervisor |
| Guardrails |
| Judges online |
| MCP Tool Router |
| RAG |
| Memory / Checkpoint |
| Langfuse Telemetry |
+-----------+------------+
|
v
+------------------------+
| Langfuse |
|------------------------|
| Traces |
| Spans |
| Generations |
| Metadata |
| Usage / Tokens |
+-----------+------------+
|
v
+------------------------+
| agent_framework_ |
| evaluator |
|------------------------|
| Collectors |
| Identity Resolver |
| Conversation Records |
| LLM Judge |
| VLoop analytics |
| Repository Oracle |
| Legacy Exporter |
| API / CLI |
+-----------+------------+
|
v
+------------------------+
| Oracle ADB |
|------------------------|
| EVALUATION_RUN |
| EVALUATION_ITEM |
| EVALUATION_RESULT |
| EVALUATION_FINDING |
| EVALUATION_PROGRESS |
| EVALUATION_METRIC |
+-----------+------------+
|
v
+------------------------+
| Output |
|------------------------|
| TXT.GZ legacy |
| API dashboard |
| Langfuse scores |
+------------------------+
```
---
## 5. Solution components
### 5.1 CLI
Main file:
```text
evaluator/cli.py
```
Responsible for exposing commands such as:
```bash
python -m evaluator.cli init-db
python -m evaluator.cli show-config
python -m evaluator.cli run --source langfuse
python -m evaluator.cli run-agents --source langfuse
python -m evaluator.cli runs
python -m evaluator.cli progress <run_id>
```
The CLI is the main way to operate the evaluator in batch mode.
---
### 5.2 API
Main file:
```text
evaluator/api/main.py
```
Exposes HTTP endpoints to query progress, runs, and results.
Expected examples:
```text
GET /health
GET /runs
GET /runs/{run_id}/progress
GET /runs/{run_id}/results
GET /runs/{run_id}/findings
```
The API allows you to build a simple graphical interface or integrate the evaluator with other systems.
---
### 5.3 EvaluationEngine
Main file:
```text
evaluator/engine.py
```
It is the central orchestrator of the evaluator.
Responsibilities:
1. create a new evaluation run (`EVALUATION_RUN`);
2. choose the collector according to the `source`;
3. collect conversations;
4. apply sampling by agent;
5. insert items into `EVALUATION_ITEM`;
6. process each item;
7. call the LLM Judge;
8. save trace result;
9. run session evaluation;
10. save session result;
11. export legacy file;
12. mark final execution status;
13. issue progress events.
Simplified flow:
```text
run_agent()
↓
collector.collect()
↓
repository.insert_items()
↓
_process()
↓
judge.judge_trace()
↓
repository.save_trace_result()
↓
judge.judge_sessions()
↓
repository.save_session_result()
↓
export_legacy_txt_gz()
```
---
### 5.4 Collectors
Directory:
```text
evaluator/collectors/
```
Collectors are responsible for fetching conversations from an external source and converting them to `ConversationRecord`.
Typical collectors:
```text
evaluator/collectors/langfuse.py
evaluator/collectors/agent_framework.py
evaluator/collectors/mock.py
evaluator/collectors/base.py
```
#### LangfuseCollector
This is the main collector.
Responsibilities:
- search for traces in Langfuse;
- filter by period;
- filter by agent/alias;
- retrieve trace details;
- extract input/output;
- reconstruct messages;
- collect metadata;
- apply `identity.yaml`;
- assemble canonical `BusinessContext`;
- fill in `ConversationRecord`.
The collector must normalize data so that the exporter does not need to know Langfuse's internal details.
---
### 5.5 Identity Resolver
Recommended directory:
```text
evaluator/identity/
```
Main file:
```text
evaluator/identity/resolver.py
```
The evaluator must use the same identity concept as `agent_framework_oci`, based on the file:
```text
configs/identity.yaml
```
The function of `identity.yaml` is to map variable input fields to a canonical model:
```text
customer_key
contract_key
interaction_key
account_key
resource_key
session_key
```
Conceptual example:
```yaml
identity:
version: 2
keys:
customer_key:
sources:
- business_context.customer_key
- metadata.customer_key
- user_id
contract_key:
sources:
- business_context.contract_key
- metadata.contract_key
interaction_key:
sources:
- business_context.interaction_key
- metadata.ura_call_id
- metadata.message_id
- message_id
session_key:
sources:
- business_context.session_key
- session_id
- conversation_key
```
With this, the evaluator is not directly tied to fields such as `ura_call_id`, `call_id`, `message_id` or `interaction_key`. It resolves everything to `interaction_key`.
---
### 5.6 Models
Main file:
```text
evaluator/core/models.py
```
Defines the core objects of the evaluator.
Main models:
```python
class ConversationRecord
class ConversationMessage
class TraceJudgeResult
class SessionJudgeResult
class CombinedJudgeResult
class RunStatus
class ItemStatus
```
#### ConversationRecord
Represents an evaluated conversation or turn.
Common fields:
```text
trace_id
session_id
message_id
agent_id
channel
input_text
output_text
messages
metadata
raw
```
The `metadata` field must contain normalized data:
```text
business_context
uraCallId
channelId
messageId
promptLength
```
The `raw` field keeps the original payload for auditing and fallback.
---
### 5.7 LLM Judge
Main file:
```text
evaluator/judges/llm_judge.py
```
Main class:
```python
TIMStyleLLMJudge
```
Responsibilities:
- load evaluation prompts;
- set up trace prompt;
- set up session prompt;
- call LLM via configured client;
- interpret JSON response;
- return `TraceJudgeResult` and `SessionJudgeResult`.
The judge evaluates metrics such as:
```text
judgeScore
accuracyScore
alucinationScore
inferredCsiScore
resolution
conversationPrecision
rationale
```
The judge must be LLM-based, not deterministic.
---
### 5.8 Prompts
Directory:
```text
evaluator/prompts/
```
Expected files:
```text
trace_judge_prompt.md
session_judge_prompt.md
loader.py
```
The trace prompt evaluates an individual response.
The session prompt evaluates the conversation grouped by `session_id`.
Example of expected LLM output for trace:
```json
{
"judgeScore": 0.8,
"accuracyScore": 0.9,
"alucinationScore": 0.1,
"rationale": "A response that is relevant to the context and based on available data."
}
```
Example of expected output for session:
```json
{
"inferredCsiScore": 0.5,
"resolution": 1,
"conversationPrecision": 1,
"rationale": "The conversation was resolved with consistent information."
}
```
---
### 5.9 LLM Client
Directory:
```text
evaluator/llm/
```
Typical files:
```text
evaluator/llm/client.py
evaluator/llm/oci_openai.py
```
The evaluator must use the same LLM access pattern as `agent_framework_oci`, preferably via the `oci_openai` provider.
Common variables:
```env
LLM_PROVIDER=oci_openai
OCI_GENAI_ENDPOINT=...
OCI_GENAI_MODEL_ID=...
OCI_GENAI_API_KEY=...
OCI_GENAI_COMPARTMENT_ID=...
```
The client needs to return raw text for the Judge to interpret as JSON.
---
### 5.10 Repository / Oracle Store
Directory:
```text
evaluator/persistence/
```
Main files:
```text
evaluator/persistence/oracle_store.py
evaluator/persistence/repository.py
```
`OracleStore` takes care of:
- connection with ADB/Oracle;
- wallet;
- DSN;
- schema creation/adjustment;
- thread-safe execution for asynchronous calls;
- table prefix.
The `EvaluationRepository` takes care of:
- creating runs;
- recording progress;
- inserting items;
- search for next items;
- marking an item as `PROCESSING`, `COMPLETED` or `FAILED`;
- save results;
- save findings;
- summarize run;
- list runs;
- check progress.
---
### 5.11 Legacy Exporter
Main file:
```text
evaluator/output/legacy_exporter.py
```
Generates the legacy file:
```text
output/AGENTE_<agent_id>_LLM_JUDGE_YYYYMMDD.TXT.GZ
```
Column format:
```text
judgeScore
accuracyScore
alucinationScore
promptLength
loop
inferredCsiScore
resolution
conversationPrecision
uraCallId
channelId
sessionId
messageId
```
Example:
```text
"0.8"|;"0.9"|;"0.1"|;"732"|;"0"|;"0.5"|;"1"|;"1"|;"6d7e85b0-ddd0-4f23-a372-30e754a4491a"|;"web"|;"eba23248-e038-4d33-bc2c-6465ef677d07"|;"6d7e85b0-ddd0-4f23-a372-30e754a4491a"
"TOTAL"|;"19"
```
#### promptLength
The `promptLength` field must use this priority:
1. `prompt_tokens`/ `promptTokens` /`input_tokens`/ `inputTokens` in Langfuse observations;
2. `usage.input` or `usageDetails.input`;
3. `metadata.input_size` issued by the framework;
4. fallback for text size of `input_text`, `output_text`, and `messages`.
Example:
```text
promptLength = 732
```
#### loop
The `loop field` uses the VLoop detector.
```text
0 = sem loop detectado
1 = loop detectado
```
---
### 5.12 VLoop Analytics
Main file:
```text
evaluator/analytics/vloop.py
```
Responsible for detecting conversational repetition/loop in a pattern similar to the VLoop guardrail of `agent_framework_oci`.
The function normally exposed is:
```python
vloop_flag(raw) -> int
```
It returns:
```text
0 when there is no evidence of a loop
1 when there is suspected repetition
```
---
### 5.13 Langfuse Score Publisher
Main file:
```text
evaluator/publishers/langfuse_scores.py
```
Responsible for publishing evaluation scores back to Langfuse, when enabled.
Control variable:
```env
PUBLISH_LANGFUSE_SCORES=true
```
When disabled, the evaluator only writes to the database and exports the file.
---
## 6. Directory structure
```text
agent_framework_evaluator/
├── configs/
│ ├── identity.yaml
│ └── judge/
│ └── agents.yaml
├── docs/
├── evaluator/
│ ├── __init__.py
│ ├── cli.py
│ ├── engine.py
│ ├── api/
│ │ └── main.py
│ ├── analytics/
│ │ └── vloop.py
│ ├── collectors/
│ │ ├── base.py
│ │ ├── langfuse.py
│ │ ├── agent_framework.py
│ │ └── mock.py
│ ├── config/
│ │ ├── settings.py
│ │ └── agents.py
│ ├── core/
│ │ └── models.py
│ ├── identity/
│ │ └── resolver.py
│ ├── judges/
│ │ └── llm_judge.py
│ ├── llm/
│ │ ├── client.py
│ │ └── oci_openai.py
│ ├── output/
│ │ └── legacy_exporter.py
│ ├── persistence/
│ │ ├── oracle_store.py
│ │ └── repository.py
│ ├── prompts/
│ │ ├── loader.py
│ │ ├── trace_judge_prompt.md
│ │ └── session_judge_prompt.md
│ └── publishers/
│ └── langfuse_scores.py
├── output/
├── Dockerfile
├── docker-compose.yml
├── pyproject.toml
└── README.md
```
---
## 7. Configuration
### 7.1 `.env file`
Example:
```env
# Oracle / ADB
ADB_USER=ADMIN
ADB_PASSWORD=your_password
ADB_DSN=oradb23ai_high
ADB_WALLET_DIR=/path/to/Wallet_ORADB23ai
DB_TABLE_PREFIX=AGENTFW_
# Langfuse
LANGFUSE_ENABLED=true
LANGFUSE_HOST=http://localhost:3005
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
PUBLISH_LANGFUSE_SCORES=false
# LLM
LLM_PROVIDER=oci_openai
OCI_GENAI_ENDPOINT=https://...
OCI_GENAI_MODEL_ID=...
OCI_GENAI_API_KEY=...
OCI_GENAI_COMPARTMENT_ID=...
# Evaluator
EVALUATOR_OUTPUT_DIR=output
EVALUATOR_BATCH_SIZE=10
EVALUATOR_MAX_ATTEMPTS=2
EVALUATOR_AGENTS_CONFIG=configs/judge/agents.yaml
IDENTITY_CONFIG_PATH=configs/identity.yaml
TRACE_PROMPT_PATH=evaluator/prompts/trace_judge_prompt.md
SESSION_PROMPT_PATH=evaluator/prompts/session_judge_prompt.md
```
---
### 7.2 Agent configuration
File:
```text
configs/judge/agents.yaml
```
Example:
```yaml
agents:
- agent_id: telecom_contas
enabled: true
aliases:
- telecom_contas
- billing_agent
- financeiro_agent
percentage: 1.0
- agent_id: retail_orders
enabled: true
aliases:
- retail_orders
- orders_agent
percentage: 1.0
- agent_id: financeiro_agent
enabled: true
aliases:
- financeiro_agent
percentage: 1.0
```
The `aliases` field is important because Langfuse can register the agent in different ways, for example:
```text
agent_id = telecom_contas
route = financeiro_agent
agent = financeiro_agent
```
---
### 7.3 Identity configuration
File:
```text
configs/identity.yaml
```
The evaluator must use the same pattern as the framework.
Example:
```yaml
identity:
version: 2
keys:
customer_key:
sources:
- business_context.customer_key
- metadata.customer_key
- user_id
contract_key:
sources:
- business_context.contract_key
- metadata.contract_key
interaction_key:
sources:
- business_context.interaction_key
- metadata.ura_call_id
- metadata.message_id
- message_id
session_key:
sources:
- business_context.session_key
- metadata.session_key
- session_id
- conversation_key
```
The `interaction_key` field is used to populate the `uraCallId` in the legacy export.
---
## 8. How to run
### 8.1 Install dependencies
```bash
python -m venv .venv
source .venv/bin/activate
pip install -e .
```
If you are using Conda:
```bash
conda activate py313
pip install -e .
```
---
### 8.2 Validate configuration
```bash
python -m evaluator.cli show-config
```
Expected output:
```text
{
"env_path": ".../.env",
"adb_dsn": "oradb23ai_high",
"wallet": ".../Wallet_ORADB23ai",
"langfuse": true,
"publish_langfuse_scores": false,
"llm_provider": "oci_openai",
"agents_config": "configs/judge/agents.yaml"
}
```
---
### 8.3 Create/validate schema
```bash
python -m evaluator.cli init-db
```
Expected output:
```text
{'status': 'OK', 'message': 'Evaluator schema checked/created successfully.'}
```
---
### 8.4 Run evaluation by period
```bash
python -m evaluator.cli run \
--period-start 2026-06-11T00:00:00 \
--period-end 2026-06-12T00:00:00 \
--source langfuse
```
---
### 8.5 Run evaluation by configured agents
```bash
python -m evaluator.cli run-agents --source langfuse
```
Expected output:
```text
[
{
'status': 'COMPLETED',
'run_id': '...',
'total_items': 19,
'completed_items': 19,
'failed_items': 0,
'evaluations': 19,
'avg_score': 0.72,
'agent_id': 'telecom_contas',
'output_file': 'output/AGENTE_telecom_contas_LLM_JUDGE_20260612.TXT.GZ',
'uploaded_to': None
}
]
```
---
### 8.6 Check progress
```bash
python -m evaluator.cli progress <run_id>
```
Or via API:
```bash
curl http://localhost:8001/runs/<run_id>/progress
```
---
### 8.7 View exported file
```bash
gzip -cd output/AGENTE_telecom_contas_LLM_JUDGE_20260612.TXT.GZ
```
Example of a valid line:
```text
"0.8"|;"0.9"|;"0.1"|;"732"|;"0"|;"0.5"|;"1"|;"1"|;"6d7e85b0-ddd0-4f23-a372-30e754a4491a"|;"web"|;"eba23248-e038-4d33-bc2c-6465ef677d07"|;"6d7e85b0-ddd0-4f23-a372-30e754a4491a"
"TOTAL"|;"19"
```
---
## 9. Database
### 9.1 Main tables
#### EVALUATION_RUN
Stores an evaluation run.
Main fields:
```text
RUN_ID
PERIOD_START
PERIOD_END
SOURCE
AGENT_ID
STATUS
TOTAL_ITEMS
PROCESSED_ITEMS
FAILED_ITEMS
LAST_HEARTBEAT_AT
CREATED_AT
UPDATED_AT
ERROR_MESSAGE
```
---
#### EVALUATION_ITEM
Stores each conversation/turn collected.
Main fields:
```text
ITEM_ID
RUN_ID
TRACE_ID
SESSION_ID
MESSAGE_ID
AGENT_ID
CHANNEL
STATUS
ATTEMPT_COUNT
RAW_JSON
CREATED_AT
UPDATED_AT
ERROR_MESSAGE
```
---
#### EVALUATION_RESULT
Stores trace and session results.
Main fields:
```text
RESULT_ID
RUN_ID
ITEM_ID
TRACE_ID
SESSION_ID
AGENT_ID
JUDGE_TYPE
JUDGE_NAME
JUDGE_SCORE
ACCURACY_SCORE
ALUCINATION_SCORE
INFERRED_CSI_SCORE
RESOLUTION
CONVERSATION_PRECISION
RATIONALE
RESULT_JSON
CREATED_AT
```
`JUDGE_TYPE` can be:
```text
TRACE
SESSION
```
---
#### EVALUATION_PROGRESS_EVENT
Stores execution progress events.
Stage examples:
```text
RUN_CREATED
COLLECTING
COLLECTED
SAMPLED
ITEMS_INSERTED
BATCH_STARTED
ITEM_COMPLETED
ITEM_FAILED
SESSION_JUDGE_COMPLETED
EXPORTED
COMPLETED
PARTIAL
```
---
## 10. How the codes work together
### 10.1 Complete execution flow
```text
CLI run-agents
↓
load configs/judge/agents.yaml
↓
for each enabled agent
↓
EvaluationEngine.run_agent(agent)
↓
cria EVALUATION_RUN
↓
LangfuseCollector.collect(...)
↓
IdentityResolver.resolve(...)
↓
ConversationRecord
↓
EvaluationRepository.insert_items(...)
↓
EvaluationEngine._process(run_id)
↓
TIMStyleLLMJudge.judge_trace(record)
↓
LLMClient.complete(prompt)
↓
save_trace_result(...)
↓
TIMStyleLLMJudge.judge_sessions(records)
↓
save_session_result(...)
↓
export_legacy_txt_gz(...)
↓
COMPLETED
```
---
### 10.2 Role of the collector
The collector is responsible for transforming external data into canonical data.
It must hide differences between sources such as:
```text
Langfuse
agent_framework database
mock data
```
The output must always be:
```python
ConversationRecord
```
---
### 10.3 Role of the judge
The judge receives a `ConversationRecord`, assembles a prompt, and calls the LLM.
It should not know about Oracle, Langfuse, legacy export, or API.
It only evaluates.
---
### 10.4 Role of the repository
The repository is the persistence layer.
It must not contain an evaluation business rule.
It only writes, retrieves, and updates data.
---
### 10.5 Role of the exporter
The exporter transforms persisted data into a legacy file.
It should not resolve identity in a complex way.
Ideally, it should read fields that are already normalized:
```text
metadata.business_context.interaction_key
metadata.channelId
metadata.messageId
metadata.promptLength
```
However, for resilience, it can also query `RAW_JSON` as a fallback.
---
## 11. Important design rules
### 11.1 The evaluator must not be anchored to an agent
Avoid logic like:
```python
if agent_id == "telecom_contas":
ura_call_id = metadata["ura_call_id"]
```
The correct thing to do is to use `identity.yaml`.
---
### 11.2 The exporter must not know internal details of Langfuse
Avoid excessive coupling to paths such as:
```text
raw.detail.observations[0].metadata.ura_call_id
raw.trace.input.business_context.interaction_key
```
This should be resolved in the collector.
---
### 11.3 `promptLength` should come from tokens when possible
Recommended priority:
```text
1. prompt_tokens / promptTokens
2. input_tokens / inputTokens
3. usage.input / usageDetails.input
4. metadata.input_size
5. tamanho textual de input/output/messages
```
---
### 11.4 `uraCallId` must come from BusinessContext
The legacy field `uraCallId` must be mapped to:
```text
business_context.interaction_key
```
This is the canonical name of the framework.
---
### 11.5 `sessionId` must come from BusinessContext
The legacy `sessionId` field must be mapped to:
```text
business_context.session_key
```
Not to be confused with the full composite key:
```text
default:telecom_contas:<uuid>
```
The evaluator can store the full key, but the legacy export should normally use the clean session identifier.
---
## 12. Recommended tests
### 12.1 Configuration test
```bash
python -m evaluator.cli show-config
```
Validate:
```text
ADB_DSN
Wallet
Langfuse enabled
LLM provider
Agents config
Identity config
```
---
### 12.2 Database test
```bash
python -m evaluator.cli init-db
```
Then validate tables:
```sql
select table_name
from user_tables
where table_name like 'AGENTFW_EVALUATION%';
```
---
### 12.3 Mock test
```bash
python -m evaluator.cli run --source mock
```
Use this test to validate schema, judge, and export without relying on Langfuse.
---
### 12.4 Test with Langfuse
```bash
python -m evaluator.cli run-agents --source langfuse
```
Validate:
```text
total_items > 0
completed_items > 0
failed_items = 0
evaluations > 0
output_file preenchido
```
---
### 12.5 Export test
```bash
gzip -cd output/AGENTE_telecom_contas_LLM_JUDGE_YYYYMMDD.TXT.GZ
```
Validate columns:
```text
judgeScore filled in
accuracyScore filled in
hallucinationScore filled in
promptLength greater than 0
loop 0 or 1
inferredCsiScore filled in
resolution 0 or 1
conversationPrecision 0 or 1
uraCallId filled in
channelId filled in
sessionId filled in
messageId filled in
```
---
## 13. Troubleshooting
### 13.1 `promptLength` outputs 0
Common causes:
- `find_prompt_tokens` was not included in the file;
- `promptTokens` is zeroed in Langfuse;
- `input_size` is not being traversed;
- `RAW_JSON` is coming as an unconverted string;
- old exporter is still running;
- `except Exception: pass` is masking error.
Recommended debug:
```python
print("PROMPT_LENGTH", extract_prompt_length(raw))
print("RAW_TYPE", type(raw))
print("RAW_KEYS", list(raw.keys())[:20])
```
---
### 13.2 `uraCallId` comes out empty
Common causes:
- `identity.yaml` is not being loaded;
- collector is not copying `business_context` to `metadata`;
- `interaction_key` does not exist in the trace;
- exporter does not use `business_context.interaction_key`.
Validation:
```sql
select RAW_JSON
from AGENTFW_EVALUATION_ITEM
where MESSAGE_ID = '<message_id>';
```
Search:
```text
interaction_key
ura_call_id
business_context
```
---
### 13.3 `ORA-00904 invalid identifier`
Usually indicates an old schema.
Examples already found:
```text
ORA-00904: UPDATED_AT invalid identifier
ORA-00904: REASONING invalid identifier
ORA-00904: JUDGE_TYPE invalid identifier
```
Correction:
```bash
python -m evaluator.cli init-db
```
If the table already exists without the new column,`_init_schema` needs to run `ALTER TABLE ADD` in an idempotent manner.
---
### 13.4 `ORA-00054 resource busy`
Indicates a lock on the table.
Common causes:
- API running while `init-db` tries to change schema;
- another process using the table;
- transaction open in SQL Developer.
Correction:
1. stop API/CLI;
2. close open sessions;
3. run `init-db` again.
---
### 13.5 `OCI LLM 401`
Indicates an authentication problem in the LLM.
Validate:
```env
OCI_GENAI_ENDPOINT
OCI_GENAI_MODEL_ID
OCI_GENAI_API_KEY
OCI_GENAI_COMPARTMENT_ID
```
Also confirm that the evaluator is reading the correct `.env`:
```bash
python -m evaluator.cli show-config
```
---
### 13.6 `Entity with key ${OCI_GENAI_MODEL_ID} not found`
Indicates that the literal value `${OCI_GENAI_MODEL_ID}` has reached the provider.
Common causes:
- variable not expanded;
- YAML using `${OCI_GENAI_MODEL_ID}` without interpolation;
- `.env` not loaded;
- LLM client configuration does not resolve placeholders.
Correction:
- put the real model ID in `the .env`;
- ensure interpolation in `settings.py`;
- validate with `show-config`.
---
## 14. Final validation checklist
Before considering the evaluator ready, validate:
```text
[ ] init-db executes without error
[ ] show-config displays correct .env file
[ ] Langfuse returns traces
[ ] run-agents collects items per agent
[ ] LLM Judge responds with valid JSON
[ ] EVALUATION_RESULT records TRACE and SESSION data
[ ] progress displays useful events
[ ] export TXT.GZ is generated
[ ] promptLength > 0
[ ] uraCallId populated
[ ] sessionId populated
[ ] messageId populated
[ ] loop populated with 0 or 1
[ ] file ends with TOTAL
[ ] scores can be published to Langfuse when enabled
```
---
## 15. Example of validated final result
```text
"0.8"|;"0.9"|;"0.1"|;"732"|;"0"|;"0.5"|;"1"|;"1"|;"6d7e85b0-ddd0-4f23-a372-30e754a4491a"|;"web"|;"eba23248-e038-4d33-bc2c-6465ef677d07"|;"6d7e85b0-ddd0-4f23-a372-30e754a4491a"
"0.9"|;"1"|;"0"|;"642"|;"0"|;"0.5"|;"1"|;"1"|;"5ab3ea80-7428-402f-98ec-04e7cd5327e4"|;"web"|;"eba23248-e038-4d33-bc2c-6465ef677d07"|;"5ab3ea80-7428-402f-98ec-04e7cd5327e4"
"TOTAL"|;"19"
```
This result indicates:
- Judge working;
- prompt tokens extracted correctly;
- VLoop without occurrence;
- session metrics filled in;
- canonical identity working;
- legacy export in the expected layout.
---
## 16. Executive summary
The `agent_framework_evaluator` is the batch/offline evaluation layer of the `agent_framework_oci` ecosystem.
It consumes the telemetry generated by the framework, applies a Judge LLM with evaluation rules, persists results in Oracle/ADB, generates a file, and can republish scores in Langfuse.
The correct architecture separates responsibilities:
```text
Collector normalizes data.
IdentityResolver resolves identity.
Judge evaluates conversation.
Repository persists data.
Exporter generates legacy data.
API/CLI operate the solution.
```
This makes the evaluator generic for multiple agents and avoids direct coupling to specific trace or payload formats.
Reference in New Issue View Git Blame Copy Permalink