mirror of
https://github.com/mcloh/decision-models-benchmark.git
synced 2026-11-06 06:14:11 +00:00
328ccd1c2e
- harness/evaluate.py: partição congelada com SHA-256, amostra com seed, lote 1, truncamento comum, contagem de tokens, latência (sem aquecimento), pico de RAM e GPU; teste bloqueado até o congelamento da configuração. - analysis/metrics.py: acurácia, macro F1, Brier, NLL, ECE, acurácia seletiva. - Diagnósticos por resposta (metadata.diagnostics); Laya informa opções colapsadas pelo orçamento de cabeçalho; head_max_len via DMB_LAYA_HEAD_MAX_LEN. - infra/ds/collect.py e gpu_queue --env para pilotos e rodadas. - .gitignore: de models/ só metadados (cards, configs, licenças, índices); pesos, tokenizadores grandes, runtimes, fontes clonados e caches ignorados. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
182 lines
9.1 KiB
Markdown
182 lines
9.1 KiB
Markdown
---
|
||
license: apache-2.0
|
||
library_name: transformers
|
||
pipeline_tag: text-classification
|
||
language: [multilingual, en, de, fr, es, pt, it, nl, sv, da, nb, ru, pl, tr, ar, he, fa, ur, hi, bn, ta, te, kn, ml, th, vi, id, ms, tl, ja, ko, zh, el, hu, fi, ro, sq, sl, sw, af, cy, am, hy, ka, km, my, mn, lv, is, az, jv]
|
||
tags: [laya, multilingual, mmbert, system-one, calibrated-decisions, rlcd, classification, routing, guardrails, moderation, commercial-use]
|
||
---
|
||
|
||
<p align="center">
|
||
<img src="https://huggingface.co/convaiinnovations/laya/resolve/main/assets/logo-mark.png" alt="" width="72" />
|
||
</p>
|
||
|
||
# Laya Multilingual
|
||
|
||
Non-autoregressive **System 1 decision model** covering 100+ languages. Give it a **state**
|
||
(text, email, ticket, or JSON) and **typed questions**; it returns typed answers with
|
||
probabilities in a single forward pass. No text generation, so nothing to parse and nothing to
|
||
hallucinate.
|
||
|
||
Part of the [Laya family](https://huggingface.co/convaiinnovations/laya) — **use this checkpoint
|
||
for anything that is not English.**
|
||
|
||
| checkpoint | encoder | params | context | use it for |
|
||
|---|---|---|---|---|
|
||
| [`convaiinnovations/laya`](https://huggingface.co/convaiinnovations/laya) | ModernBERT-large | 421M | 512 | English |
|
||
| **`convaiinnovations/laya-multilingual`** (this repo) | mmBERT-base | 322M | 1024 (up to 8,192) | 100+ languages, ~2x faster |
|
||
| [`convaiinnovations/laya-typed-decisions`](https://huggingface.co/convaiinnovations/laya-typed-decisions) | ModernBERT-large | 421M | 1024 | the typed-decisions workflows |
|
||
|
||
> **Long documents: `laya-multilingual` reads up to 8,192 tokens.** It ships with a 1,024-token limit that cuts long documents off, so pass `max_len=8192` for them:
|
||
>
|
||
> ```python
|
||
> import laya
|
||
> agent = laya.load("convaiinnovations/laya-multilingual")
|
||
> result = agent.predict(long_document, questions, max_len=8192)
|
||
> ```
|
||
>
|
||
> In the table below, 16 to 18 of 20 requests were answered correctly with up to about 4,000 tokens of text before them; beyond that results vary (8 to 17 of 20), so check long-document accuracy on your own data. Short inputs give identical answers with `max_len=8192`, and speed follows the input's real length, not the limit: short inputs are unchanged, and a 4,000-token input takes about 1.7 s on an Apple GPU.
|
||
|
||
<p align="center">
|
||
<img src="https://raw.githubusercontent.com/NandhaKishorM/laya/main/assets/long_context_8192.png" alt="laya-multilingual long-document accuracy by document length" width="100%" />
|
||
</p>
|
||
|
||
## Quickstart
|
||
|
||
```bash
|
||
pip install laya
|
||
```
|
||
|
||
```python
|
||
import laya
|
||
|
||
agent = laya.load("convaiinnovations/laya-multilingual")
|
||
result = agent.predict(
|
||
{"body": "मुझसे इनवॉइस 4411 के लिए दो बार शुल्क लिया गया। कृपया आज ही धनवापसी करें।"},
|
||
{"department": {"type": "choice", "instructions": "Which team should handle `body`?",
|
||
"criteria": {"billing": "invoices, payments, refunds",
|
||
"technical": "bugs and outages", "sales": "pricing"}},
|
||
"refund_requested": {"type": "noul", "instructions": "Does the sender ask for money back?"}},
|
||
)
|
||
print(result["answers"]["department"]["choice"]) # billing
|
||
```
|
||
|
||
### Let the Router choose
|
||
|
||
```python
|
||
from laya import Router
|
||
|
||
router = Router()
|
||
router.predict({"body": "I was charged twice"}, questions) # -> laya
|
||
router.predict({"body": "二重に請求されました"}, questions) # -> laya-multilingual
|
||
```
|
||
|
||
The default `Router()` keeps both `english` and this checkpoint resident, so a
|
||
mixed workload no longer swaps checkpoints on every language change. For a server, load them up
|
||
front so even the first request of each language is just a forward pass:
|
||
|
||
```python
|
||
router = Router()
|
||
router.preload(["english", "multilingual"]) # both resident; no swap at request time
|
||
```
|
||
|
||
`router.attach("multilingual", agent)` registers an `Agent` you already built, so a process that
|
||
loaded this checkpoint directly can hand it to the router instead of loading it twice.
|
||
|
||
More text reaches this checkpoint than script alone would send: plain-ASCII Spanish, Italian, Portuguese and French
|
||
(accents stripped by mail clients and ticket systems), Brazilian Portuguese support text, and any
|
||
script the router has no range for. If you already run a language-identification model, pass its
|
||
answer with `router.predict(state, questions, lang_guess=code_or_callable)`.
|
||
|
||
It also receives CJK requests that contain Latin brand names, romanized
|
||
Bangla, and Azerbaijani. On 20,000 English texts, at most 5 English sentences move, all quoting long
|
||
native-script names.
|
||
|
||
Routing is decided from the script of the input, **before** the forward pass — because the
|
||
model's confidence gives no warning when a checkpoint cannot read its input (see below).
|
||
|
||
> **If `laya.load()` hangs:** `transformers` probes for TensorFlow at import, and when TF is
|
||
> installed its abseil runtime can deadlock model construction. Run with `USE_TF=0`.
|
||
|
||
## Why this checkpoint exists
|
||
|
||
Measured across **all 51 MASSIVE languages**, intent classification with 20 options
|
||
(random = 0.050), both checkpoints answering byte-identical questions:
|
||
|
||
| | `laya` (English) | **`laya-multilingual`** |
|
||
|---|---|---|
|
||
| macro accuracy | 0.227 | **0.366** |
|
||
| macro ECE | 0.733 | **0.387** |
|
||
| languages clearing 3x random | 23 / 51 | **45 / 51** |
|
||
|
||
The English checkpoint does not degrade gracefully outside English — it collapses, and stays
|
||
confident while doing so. Khmer: **0.000 accuracy at 0.952 confidence**. Hebrew 0.060, Armenian
|
||
0.050 (exactly random), Bengali 0.080 — all reported with 0.89–0.96 confidence. Its mean
|
||
confidence never drops below 0.885 at any accuracy level, so confidence gating cannot catch it.
|
||
|
||
Per-language, this checkpoint turns near-random into usable: Arabic 0.110 → **0.400**, Bengali
|
||
0.080 → **0.290**, Azerbaijani 0.100 → **0.300**, Hindi 0.100 → **0.387**, Korean 0.110 →
|
||
**0.490**, Turkish 0.140 → **0.437**.
|
||
|
||
### XNLI (15 languages)
|
||
|
||
| | `laya` | **`laya-multilingual`** |
|
||
|---|---|---|
|
||
| English | **0.860** | 0.843 |
|
||
| 14 other languages | 0.521 | **0.731** |
|
||
|
||
### Speed — it is also the faster checkpoint
|
||
|
||
| questions per call | `laya` | **`laya-multilingual`** |
|
||
|---|---|---|
|
||
| 1 | 39.5 ms | **32.8 ms** |
|
||
| 10 | 158.6 ms (15.9 ms/q) | **72.3 ms (7.2 ms/q)** |
|
||
| 50 | 771 ms | **337 ms (6.8 ms/q)** |
|
||
|
||
103–332 questions/sec batched on one T4, despite a 256k vocabulary — the 768-dim / 22-layer
|
||
encoder is cheaper per token than 1024-dim / 28-layer, and the gap widens with batch size.
|
||
|
||
## Architecture
|
||
|
||
- **Backbone** mmBERT-base (307M, bidirectional, 22 layers, hidden 768, 256k vocab) + a decision
|
||
head trained from scratch: 2 transformer layers, an option-marker scorer, and an act/escalate
|
||
head. 322M total.
|
||
- **Option markers** every option is scored at its own `[MASK]` token, then softmaxed over that
|
||
question's options — so the answer space is defined per request, with no retraining.
|
||
- **Budget** 1024 tokens per question, of which 256 go to the question and its options.
|
||
- Trained from scratch with RLCD: 15,987 updates, 4 epochs, ~4.97 h.
|
||
|
||
## Limits
|
||
|
||
- **Ships uncalibrated.** `temperature = [1.0, 1.0, 1.0]` with no per-option-count buckets. It is
|
||
systematically over-confident (mean confidence 0.75–0.83 against much lower accuracy). Refitting
|
||
one temperature per (question type, option count) on held-out data moves mean ECE
|
||
**0.314 → 0.106**. Do this on your own data before trusting the probabilities.
|
||
- **Weaker on English** than the English checkpoint: 0.619 vs 0.684 macro across English suites.
|
||
Route rather than replace.
|
||
- **Near chance on typed-decisions zero-shot** — 0.342, against a 0.318 random and 0.461
|
||
majority-class baseline. Fine-tune for a specific workflow; that is where the capability comes
|
||
from.
|
||
- **Keep `choice` questions under ~20 options.** Options share the fixed 256-token head budget, so
|
||
a very large label space leaves only a few tokens per label and accuracy falls off sharply.
|
||
- Low-resource languages are weak, not fixed: Swahili 0.210, Tamil 0.250, Amharic 0.110.
|
||
- Ordinal `score` questions are the weakest primitive (SST-5 0.282), and this checkpoint has a
|
||
**measured position bias on them**: it rarely picks the first-listed level, in any language,
|
||
including English (0 of 290 in one independent run,
|
||
[#131](https://github.com/NandhaKishorM/laya/issues/131)). For English score questions use the
|
||
English checkpoint. For other languages, validate score outputs on your own data first.
|
||
- **`noul` can under-report "true" here.** On a clearly positive input, one measurement put
|
||
`P(true)` at about 0.5 while the negative case was correctly near 0
|
||
([#156](https://github.com/NandhaKishorM/laya/issues/156)). If `noul` answers look weak, the same
|
||
question as a two-option `choice` with neutral keys (`A` / `B`) and yes/no descriptions is a
|
||
useful check.
|
||
|
||
## Links
|
||
|
||
- **Docs** https://nandhakishorm.github.io/laya/
|
||
- **Hub / family** https://huggingface.co/convaiinnovations/laya
|
||
- **GitHub** https://github.com/NandhaKishorM/laya · full benchmark data on the `research` branch
|
||
- **PyPI** https://pypi.org/project/laya/
|
||
- **Demo** https://huggingface.co/spaces/convaiinnovations/laya-demo
|
||
|
||
Apache 2.0 · Convai Innovations
|