mcloh/decision-models-benchmark
1
0
Fork 0
mirror of https://github.com/mcloh/decision-models-benchmark.git synced 2026-11-05 22:04:12 +00:00
Code Issues Packages Projects Releases Wiki Activity

Calibração e congelamento da configuração (tarefas 60–62)

Browse Source 
- config/calibration.json: temperatura e limiar de abstenção por candidato e
  dispositivo, ajustados só na partição de calibração.
- config/frozen.json: hashes de configuração, calibração, partições,
  adaptadores, contrato e ambientes; libera a partição de teste.
- Q6: lote de produção = 1; tarefa 65 não se aplica neste ciclo (D4).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Marcos Lohmann
2026-10-07 18:54:20 -03:00
parent 8e5e78b841
commit d4564d9c1d
19 changed files with 7338 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
reports/calibration/README.md Normal file
View File

@@ -0,0 +1,14 @@
# Resultados — partição `calibration`
Gerado por `infra/ds/collect.py`. Lote 1; latências sem os exemplos de aquecimento; probabilidades sem calibração.
<!-- TABELA:INICIO -->
| Candidato | Disp. | Precisão | n | Acurácia | Macro F1 | ECE (bruto) | p50 ms | p95 ms | p99 ms | Carga s | RAM pico MiB | GPU pico MiB | Erros | Truncados |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| gliner | cpu | float32 | 1127 | 54.5% | 54.2% | 0.077 | 257.1 | 493.9 | 601.4 | 20.1 | 3240 | — | 0 | 0 |
| gliner | cuda | float32 | 1127 | 54.5% | 54.2% | 0.077 | 18.7 | 31.6 | 33.5 | 21.6 | 3252 | 1783 | 0 | 0 |
| laya | cpu | float32 | 1127 | 50.5% | 52.9% | 0.330 | 162.6 | 209.8 | 269.9 | 9.3 | 2681 | — | 0 | 0 |
| laya | cuda | float32 | 1127 | 50.8% | 53.3% | 0.326 | 21.5 | 22.8 | 24.0 | 9.9 | 2787 | 1811 | 0 | 0 |
| rizzo_flow | cuda | gguf-q8_0 | 1127 | 83.3% | 86.5% | 0.058 | 112.3 | 179.3 | 183.5 | 5.3 | 4528 | 5115 | 0 | 0 |
| semif | cuda | bfloat16 | 1127 | 85.8% | 88.1% | 0.055 | 117.2 | 184.2 | 186.8 | 7.9 | 9030 | 8513 | 0 | 0 |
<!-- TABELA:FIM -->

1127
reports/calibration/predictions-gliner-cpu.jsonl Normal file
View File

File diff suppressed because it is too large Load Diff

1127
reports/calibration/predictions-gliner-cuda.jsonl Normal file
View File

File diff suppressed because it is too large Load Diff

1127
reports/calibration/predictions-laya-cpu.jsonl Normal file
View File

File diff suppressed because it is too large Load Diff

1127
reports/calibration/predictions-laya-cuda.jsonl Normal file
View File

File diff suppressed because it is too large Load Diff

1127
reports/calibration/predictions-rizzo_flow-cuda.jsonl Normal file
View File

File diff suppressed because it is too large Load Diff

1127
reports/calibration/predictions-semif-cuda.jsonl Normal file
View File

File diff suppressed because it is too large Load Diff

58
reports/calibration/summary-gliner-cpu.json Normal file
View File

@@ -0,0 +1,58 @@
{
"candidate": {
"name": "gliner",
"model_id": "gliner@a35a0cd3b7a0",
"revision": "a35a0cd3b7a0f00f2effc576f454cd48fa98aa5f",
"adapter_version": "0.2.0",
"device": "cpu",
"context_limit": 4096,
"probabilities_source": "native"
},
"precision": "float32",
"device": "cpu",
"split": "calibration",
"examples": 1127,
"seed": 20261007,
"warmup": 5,
"batch_size": 1,
"state_budget_tokens": 512,
"process_guard": "ok",
"load_seconds": 20.09,
"warmup_latency_ms": [
305.3,
321.6,
329.3,
504.9,
161.4
],
"latency_ms": {
"n": 1122,
"p50": 257.1435335000558,
"p95": 493.89709470000014,
"p99": 601.4209299699689,
"mean": 280.12477045187006
},
"errors": {},
"truncated": 0,
"with_diagnostics": 0,
"operational_env": {},
"state_tokens_max": 125,
"rss_peak_mib": 3239.859375,
"quality": {
"n": 1127,
"errors": 0,
"accuracy": 0.5448092280390417,
"macro_f1": 0.5418404701263009,
"macro_precision": 0.6809469392064199,
"macro_recall": 0.5329382759578066
},
"calibration_raw": {
"brier": 0.6104995008128414,
"nll": 1.3204663491307895,
"ece": 0.07668260121420074
},
"oci": {
"region": "us-chicago-1",
"run_id": "20261007T212521Z-gliner-evaluate"
}
}

60
reports/calibration/summary-gliner-cuda.json Normal file
View File

@@ -0,0 +1,60 @@
{
"candidate": {
"name": "gliner",
"model_id": "gliner@a35a0cd3b7a0",
"revision": "a35a0cd3b7a0f00f2effc576f454cd48fa98aa5f",
"adapter_version": "0.2.0",
"device": "cuda",
"context_limit": 4096,
"probabilities_source": "native"
},
"precision": "float32",
"device": "cuda",
"split": "calibration",
"examples": 1127,
"seed": 20261007,
"warmup": 5,
"batch_size": 1,
"state_budget_tokens": 512,
"process_guard": "ok",
"load_seconds": 21.56,
"warmup_latency_ms": [
224.3,
114.1,
104.7,
271.6,
18.6
],
"latency_ms": {
"n": 1122,
"p50": 18.674051499999678,
"p95": 31.63386759999014,
"p99": 33.51271131999595,
"mean": 21.56757115775458
},
"errors": {},
"truncated": 0,
"with_diagnostics": 0,
"operational_env": {},
"state_tokens_max": 125,
"rss_peak_mib": 3251.8671875,
"gpu_mem_peak_mib": 1783.0,
"gpu_util_mean_pct": 42.40659340659341,
"quality": {
"n": 1127,
"errors": 0,
"accuracy": 0.5448092280390417,
"macro_f1": 0.5418404701263009,
"macro_precision": 0.6809469392064199,
"macro_recall": 0.5329382759578066
},
"calibration_raw": {
"brier": 0.6104993145632631,
"nll": 1.320465969739849,
"ece": 0.07668256169926013
},
"oci": {
"region": "sa-saopaulo-1",
"run_id": "20261007T214834Z-gliner-evaluate"
}
}

58
reports/calibration/summary-laya-cpu.json Normal file
View File

@@ -0,0 +1,58 @@
{
"candidate": {
"name": "laya",
"model_id": "laya@1720e3e3357c",
"revision": "1720e3e3357cfe1e281542e223f8273b0890ca34",
"adapter_version": "0.2.0",
"device": "cpu",
"context_limit": 1024,
"probabilities_source": "native"
},
"precision": "float32",
"device": "cpu",
"split": "calibration",
"examples": 1127,
"seed": 20261007,
"warmup": 5,
"batch_size": 1,
"state_budget_tokens": 512,
"process_guard": "ok",
"load_seconds": 9.35,
"warmup_latency_ms": [
252.7,
284.6,
270.8,
284.0,
153.1
],
"latency_ms": {
"n": 1122,
"p50": 162.58618700001648,
"p95": 209.81496149999828,
"p99": 269.85545572001797,
"mean": 156.04856365508033
},
"errors": {},
"truncated": 0,
"with_diagnostics": 0,
"operational_env": {},
"state_tokens_max": 100,
"rss_peak_mib": 2681.41015625,
"quality": {
"n": 1127,
"errors": 0,
"accuracy": 0.5048802129547472,
"macro_f1": 0.5293731376193379,
"macro_precision": 0.5322309316552073,
"macro_recall": 0.5716537982207718
},
"calibration_raw": {
"brier": 0.7938067217353456,
"nll": 2.800574273404074,
"ece": 0.3295974367816615
},
"oci": {
"region": "us-chicago-1",
"run_id": "20261007T212448Z-laya-evaluate"
}
}

60
reports/calibration/summary-laya-cuda.json Normal file
View File

@@ -0,0 +1,60 @@
{
"candidate": {
"name": "laya",
"model_id": "laya@1720e3e3357c",
"revision": "1720e3e3357cfe1e281542e223f8273b0890ca34",
"adapter_version": "0.2.0",
"device": "cuda",
"context_limit": 1024,
"probabilities_source": "native"
},
"precision": "float32",
"device": "cuda",
"split": "calibration",
"examples": 1127,
"seed": 20261007,
"warmup": 5,
"batch_size": 1,
"state_budget_tokens": 512,
"process_guard": "ok",
"load_seconds": 9.86,
"warmup_latency_ms": [
262.3,
24.2,
23.0,
23.0,
28.0
],
"latency_ms": {
"n": 1122,
"p50": 21.54373249999253,
"p95": 22.82198069998458,
"p99": 23.95628751000088,
"mean": 21.440272945632913
},
"errors": {},
"truncated": 0,
"with_diagnostics": 0,
"operational_env": {},
"state_tokens_max": 100,
"rss_peak_mib": 2786.82421875,
"gpu_mem_peak_mib": 1811.0,
"gpu_util_mean_pct": 21.651515151515152,
"quality": {
"n": 1127,
"errors": 0,
"accuracy": 0.5075421472937001,
"macro_f1": 0.5328213000411428,
"macro_precision": 0.5349830494378608,
"macro_recall": 0.5748388336100538
},
"calibration_raw": {
"brier": 0.7921588792291862,
"nll": 2.774820648363843,
"ece": 0.32625944117099553
},
"oci": {
"region": "sa-saopaulo-1",
"run_id": "20261007T214316Z-laya-evaluate"
}
}

60
reports/calibration/summary-rizzo_flow-cuda.json Normal file
View File

@@ -0,0 +1,60 @@
{
"candidate": {
"name": "rizzo_flow",
"model_id": "rizzo_flow@55633c8cbd2b",
"revision": "55633c8cbd2b826bd3eefdeb05310450996649df",
"adapter_version": "0.2.0",
"device": "cuda",
"context_limit": 2048,
"probabilities_source": "native"
},
"precision": "gguf-q8_0",
"device": "cuda",
"split": "calibration",
"examples": 1127,
"seed": 20261007,
"warmup": 5,
"batch_size": 1,
"state_budget_tokens": 512,
"process_guard": "ok",
"load_seconds": 5.27,
"warmup_latency_ms": [
136.3,
142.9,
153.5,
173.8,
76.4
],
"latency_ms": {
"n": 1122,
"p50": 112.3337810000038,
"p95": 179.28425529999572,
"p99": 183.46083652000314,
"mean": 124.20437254812933
},
"errors": {},
"truncated": 0,
"with_diagnostics": 0,
"operational_env": {},
"state_tokens_max": 129,
"rss_peak_mib": 4528.34375,
"gpu_mem_peak_mib": 5115.0,
"gpu_util_mean_pct": 73.34859154929578,
"quality": {
"n": 1127,
"errors": 0,
"accuracy": 0.8331854480922803,
"macro_f1": 0.8652254669700142,
"macro_precision": 0.8424194469837629,
"macro_recall": 0.9124800045182142
},
"calibration_raw": {
"brier": 0.25364558210635535,
"nll": 0.5446881448991246,
"ece": 0.057692546501372156
},
"oci": {
"region": "sa-saopaulo-1",
"run_id": "20261007T213634Z-rizzo_flow-evaluate"
}
}

60
reports/calibration/summary-semif-cuda.json Normal file
View File

@@ -0,0 +1,60 @@
{
"candidate": {
"name": "semif",
"model_id": "semif@851bf6e806ef",
"revision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"adapter_version": "0.2.0",
"device": "cuda",
"context_limit": 8192,
"probabilities_source": "native"
},
"precision": "bfloat16",
"device": "cuda",
"split": "calibration",
"examples": 1127,
"seed": 20261007,
"warmup": 5,
"batch_size": 1,
"state_budget_tokens": 512,
"process_guard": "ok",
"load_seconds": 7.9,
"warmup_latency_ms": [
436.0,
138.5,
142.0,
178.2,
76.3
],
"latency_ms": {
"n": 1122,
"p50": 117.22612299996626,
"p95": 184.17156100000938,
"p99": 186.84284949001267,
"mean": 125.846100245989
},
"errors": {},
"truncated": 0,
"with_diagnostics": 0,
"operational_env": {},
"state_tokens_max": 101,
"rss_peak_mib": 9030.08203125,
"gpu_mem_peak_mib": 8513.0,
"gpu_util_mean_pct": 87.37113402061856,
"quality": {
"n": 1127,
"errors": 0,
"accuracy": 0.8580301685891748,
"macro_f1": 0.8813620791280374,
"macro_precision": 0.8798441924161988,
"macro_recall": 0.8952594286017203
},
"calibration_raw": {
"brier": 0.2230350259655119,
"nll": 0.47760006324176385,
"ece": 0.05521030938135974
},
"oci": {
"region": "sa-saopaulo-1",
"run_id": "20261007T212831Z-semif-evaluate"
}
}