adding: OUTPUT_FILENAME_PATTERN

This commit is contained in:
2026-09-09 19:05:12 -03:00
parent b2a368d71a
commit b0613d54ab
104 changed files with 71 additions and 5 deletions

BIN
.DS_Store vendored

Binary file not shown.

3
.env.example Normal file → Executable file
View File

@@ -44,6 +44,8 @@ AGENTS_CONFIG_PATH=configs/judge/agents.yaml
TRACE_PROMPT_PATH=configs/judge/trace_metrics.yaml
SESSION_PROMPT_PATH=configs/judge/session_metrics.yaml
OUTPUT_DIR=output
# Available placeholders: {agent_id}, {run_id}, {timestamp} (YYYYMMDD_HHMMSS)
OUTPUT_FILENAME_PATTERN=AGENTE_{agent_id}_LLM_JUDGE_{timestamp}.TXT.GZ
BATCH_SIZE=50
MAX_ATTEMPTS=3
CONTEXT_TURN_LIMIT=6
@@ -58,4 +60,3 @@ GOOGLE_APPLICATION_CREDENTIALS=configs/GCP_ACCESS_KEY.json
# Langfuse credentials are declared in one external YAML file.
# In production, mount it from a Kubernetes Secret.
LANGFUSE_CREDENTIALS_CONFIG_PATH=configs/langfuse/langfuse_credentials.yaml

0
Dockerfile Normal file → Executable file
View File

0
Documentation/llm-as-a-judge.pdf Normal file → Executable file
View File

12
README.en-US.md Normal file → Executable file
View File

@@ -688,7 +688,16 @@ GOOGLE_APPLICATION_CREDENTIALS=/path/GCP_ACCESS_KEY.json
## 12. Output file
Pattern:
The name is configured directly in `.env`:
```env
OUTPUT_FILENAME_PATTERN=AGENTE_{agent_id}_LLM_JUDGE_{timestamp}.TXT.GZ
```
Available placeholders are `{agent_id}`, `{run_id}`, and `{timestamp}`. The
`timestamp` uses the `YYYYMMDD_HHMMSS` format in the São Paulo time zone.
Resulting default pattern:
```text
AGENTE_{agentId}_LLM_JUDGE_{YYYYMMDD}_{HHMMSS}.TXT.GZ
@@ -1463,6 +1472,7 @@ python -m evaluator.cli progress <RUN_ID> --events 100
| `JUDGE_GCS_BUCKET` | bucket override |
| `GOOGLE_APPLICATION_CREDENTIALS` | GCP credentials |
| `OUTPUT_DIR` | temporary export directory |
| `OUTPUT_FILENAME_PATTERN` | configurable output filename template |
---

12
README.md Normal file → Executable file
View File

@@ -688,7 +688,16 @@ GOOGLE_APPLICATION_CREDENTIALS=/path/GCP_ACCESS_KEY.json
## 12. Arquivo de saída
Padrão:
O nome é configurado diretamente no `.env`:
```env
OUTPUT_FILENAME_PATTERN=AGENTE_{agent_id}_LLM_JUDGE_{timestamp}.TXT.GZ
```
Os placeholders disponíveis são `{agent_id}`, `{run_id}` e `{timestamp}`. O
`timestamp` é gerado no formato `YYYYMMDD_HHMMSS`, no fuso de São Paulo.
Padrão resultante:
```text
AGENTE_{agentId}_LLM_JUDGE_{YYYYMMDD}_{HHMMSS}.TXT.GZ
@@ -1478,6 +1487,7 @@ Abra `http://localhost:8080/ui`. Em Kubernetes, prefira `kubectl port-forward sv
| `JUDGE_GCS_BUCKET` | override de bucket |
| `GOOGLE_APPLICATION_CREDENTIALS` | credencial GCP |
| `OUTPUT_DIR` | diretório temporário de export |
| `OUTPUT_FILENAME_PATTERN` | template configurável do nome do arquivo de saída |
---

BIN
configs/.DS_Store vendored

Binary file not shown.

0
configs/identity.yaml Normal file → Executable file
View File

0
configs/judge/agents.yaml Normal file → Executable file
View File

0
configs/judge/session_metrics.yaml Normal file → Executable file
View File

0
configs/judge/storage.yaml Normal file → Executable file
View File

0
configs/judge/trace_metrics.yaml Normal file → Executable file
View File

0
configs/langfuse/langfuse_credentials.example.yaml Normal file → Executable file
View File

0
configs/llm_profiles/llm_profiles.yaml Normal file → Executable file
View File

BIN
deploy/.DS_Store vendored

Binary file not shown.

0
deploy/README.md Normal file → Executable file
View File

1
deploy/env/evaluator.env.example vendored Normal file → Executable file
View File

@@ -17,6 +17,7 @@ ENABLE_GCS_UPLOAD=true
JUDGE_GCS_BUCKET=
STORAGE_CONFIG_PATH=configs/judge/storage.yaml
GOOGLE_APPLICATION_CREDENTIALS=/var/secrets/google/GCP_ACCESS_KEY.json
OUTPUT_FILENAME_PATTERN=AGENTE_{agent_id}_LLM_JUDGE_{timestamp}.TXT.GZ
# Langfuse per-agent credentials are supplied by the mounted YAML file, not env names.
LANGFUSE_CREDENTIALS_CONFIG_PATH=/var/secrets/langfuse/langfuse_credentials.yaml

BIN
deploy/k8s/.DS_Store vendored

Binary file not shown.

1
deploy/k8s/base/configmap.yaml Normal file → Executable file
View File

@@ -20,6 +20,7 @@ data:
BATCH_SIZE: "50"
MAX_ATTEMPTS: "3"
ENABLE_GCS_UPLOAD: "true"
OUTPUT_FILENAME_PATTERN: "AGENTE_{agent_id}_LLM_JUDGE_{timestamp}.TXT.GZ"
STORAGE_CONFIG_PATH: "configs/judge/storage.yaml"
ADB_TABLE_PREFIX: "AGENTFW"
ADB_WALLET_LOCATION: "/opt/oracle/wallet"

0
deploy/k8s/base/cronjob.yaml Normal file → Executable file
View File

0
deploy/k8s/base/init-db-job.yaml Normal file → Executable file
View File

0
deploy/k8s/base/kustomization.yaml Normal file → Executable file
View File

0
deploy/k8s/base/monitor-deployment.yaml Normal file → Executable file
View File

0
deploy/k8s/base/monitor-service.yaml Normal file → Executable file
View File

0
deploy/k8s/base/namespace.yaml Normal file → Executable file
View File

0
deploy/k8s/base/secret-template.yaml Normal file → Executable file
View File

0
deploy/k8s/base/serviceaccount.yaml Normal file → Executable file
View File

0
deploy/k8s/overlays/dev/kustomization.yaml Normal file → Executable file
View File

0
deploy/k8s/overlays/hml/kustomization.yaml Normal file → Executable file
View File

0
deploy/k8s/overlays/prod/kustomization.yaml Normal file → Executable file
View File

0
deploy/langfuse/langfuse_credentials.example.yaml Normal file → Executable file
View File

0
deploy/scripts/build-push.sh Normal file → Executable file
View File

0
deploy/scripts/common.sh Normal file → Executable file
View File

0
deploy/scripts/create-secrets.sh Normal file → Executable file
View File

0
deploy/scripts/deploy.sh Normal file → Executable file
View File

0
deploy/scripts/run-now.sh Normal file → Executable file
View File

0
deploy/scripts/status.sh Normal file → Executable file
View File

0
deploy/scripts/undeploy.sh Normal file → Executable file
View File

BIN
evaluator/.DS_Store vendored

Binary file not shown.

0
evaluator/__init__.py Normal file → Executable file
View File

0
evaluator/__pycache__/__init__.cpython-313.pyc Normal file → Executable file
View File

0
evaluator/__pycache__/cli.cpython-313.pyc Normal file → Executable file
View File

0
evaluator/__pycache__/engine.cpython-313.pyc Normal file → Executable file
View File

0
evaluator/__pycache__/time_utils.cpython-313.pyc Normal file → Executable file
View File

0
evaluator/analytics/__init__.py Normal file → Executable file
View File

View File

0
evaluator/analytics/__pycache__/vloop.cpython-313.pyc Normal file → Executable file
View File

0
evaluator/analytics/vloop.py Normal file → Executable file
View File

0
evaluator/cli.py Normal file → Executable file
View File

View File

0
evaluator/collectors/__pycache__/base.cpython-313.pyc Normal file → Executable file
View File

View File

View File

0
evaluator/collectors/__pycache__/mock.cpython-313.pyc Normal file → Executable file
View File

0
evaluator/collectors/agent_framework.py Normal file → Executable file
View File

0
evaluator/collectors/base.py Normal file → Executable file
View File

0
evaluator/collectors/langfuse.py Normal file → Executable file
View File

0
evaluator/collectors/langfuse_credentials.py Normal file → Executable file
View File

0
evaluator/collectors/mock.py Normal file → Executable file
View File

0
evaluator/config/__pycache__/agents.cpython-313.pyc Normal file → Executable file
View File

0
evaluator/config/__pycache__/settings.cpython-313.pyc Normal file → Executable file
View File

0
evaluator/config/agents.py Normal file → Executable file
View File

4
evaluator/config/settings.py Normal file → Executable file
View File

@@ -71,6 +71,10 @@ class Settings(BaseSettings):
trace_prompt_path: str = Field(default="configs/judge/trace_metrics.yaml", validation_alias="TRACE_PROMPT_PATH")
session_prompt_path: str = Field(default="configs/judge/session_metrics.yaml", validation_alias="SESSION_PROMPT_PATH")
output_dir: str = Field(default="output", validation_alias="OUTPUT_DIR")
output_filename_pattern: str = Field(
default="AGENTE_{agent_id}_LLM_JUDGE_{timestamp}.TXT.GZ",
validation_alias="OUTPUT_FILENAME_PATTERN",
)
batch_size: int = Field(default=50, validation_alias="BATCH_SIZE")
max_attempts: int = Field(default=3, validation_alias="MAX_ATTEMPTS")
context_turn_limit: int = Field(default=6, validation_alias="CONTEXT_TURN_LIMIT")

0
evaluator/core/__pycache__/models.cpython-313.pyc Normal file → Executable file
View File

0
evaluator/core/models.py Normal file → Executable file
View File

0
evaluator/engine.py Normal file → Executable file
View File

View File

0
evaluator/identity/resolver.py Normal file → Executable file
View File

0
evaluator/judges/__pycache__/llm_judge.cpython-313.pyc Normal file → Executable file
View File

0
evaluator/judges/llm_judge.py Normal file → Executable file
View File

0
evaluator/llm/__pycache__/client.cpython-313.pyc Normal file → Executable file
View File

View File

0
evaluator/llm/client.py Normal file → Executable file
View File

0
evaluator/llm/profile_resolver.py Normal file → Executable file
View File

View File

27
evaluator/output/legacy_exporter.py Normal file → Executable file
View File

@@ -106,11 +106,36 @@ def _session_loop_map(rows) -> dict[str, int]:
return {sid: max_consecutive_intent_streak(items) for sid, items in grouped.items()}
def build_output_filename(
agent_id: str,
run_id: str,
now_brt: datetime,
pattern: str | None = None,
) -> str:
"""Build the export filename from the environment-configurable template."""
template = pattern or settings.output_filename_pattern
try:
filename = template.format(
agent_id=agent_id,
run_id=run_id,
timestamp=now_brt.strftime("%Y%m%d_%H%M%S"),
)
except KeyError as exc:
raise ValueError(
f"Unknown placeholder {exc} in OUTPUT_FILENAME_PATTERN. "
"Supported placeholders: {agent_id}, {run_id}, {timestamp}."
) from exc
if not filename or Path(filename).name != filename or filename in {".", ".."}:
raise ValueError("OUTPUT_FILENAME_PATTERN must produce a filename without directories")
return filename
def export_legacy_txt_gz(repo: EvaluationRepository, run_id: str, agent_id: str) -> Path:
output_dir = settings.path(settings.output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
now_brt = datetime.now(ZoneInfo("America/Sao_Paulo"))
path = output_dir / f"AGENTE_{agent_id}_LLM_JUDGE_{now_brt.strftime('%Y%m%d_%H%M%S')}.TXT.GZ"
path = output_dir / build_output_filename(agent_id, run_id, now_brt)
with repo.store.connect() as conn:
cur = conn.cursor()

View File

View File

0
evaluator/persistence/oracle_store.py Normal file → Executable file
View File

0
evaluator/persistence/repository.py Normal file → Executable file
View File

0
evaluator/prompts/__pycache__/loader.cpython-313.pyc Normal file → Executable file
View File

0
evaluator/prompts/loader.py Normal file → Executable file
View File

View File

View File

View File

0
evaluator/publishers/gcs_export.py Normal file → Executable file
View File

0
evaluator/publishers/langfuse_job_audit.py Normal file → Executable file
View File

0
evaluator/publishers/langfuse_scores.py Normal file → Executable file
View File

0
evaluator/time_utils.py Normal file → Executable file
View File

0
evaluator_monitor/__init__.py Normal file → Executable file
View File

0
evaluator_monitor/__pycache__/__init__.cpython-313.pyc Normal file → Executable file
View File

0
evaluator_monitor/__pycache__/app.cpython-313.pyc Normal file → Executable file
View File

0
evaluator_monitor/app.py Normal file → Executable file
View File

0
pyproject.toml Normal file → Executable file
View File

0
tests/__init__.py Normal file → Executable file
View File

0
tests/__pycache__/__init__.cpython-313.pyc Normal file → Executable file
View File

View File

View File

Some files were not shown because too many files have changed in this diff Show More