Laya, the Open-Source System 1 Decision Model: 421M Parameters, One 33 ms Forward Pass

On September 18, 2026, an independent researcher named Nandakishor M, working under the banner of Convai Innovations, pushed a repository called laya to GitHub. Five days later it had crossed 19,000 stars, and the model weights behind it sat near the top of the Hugging Face trending list. Laya is a non-autoregressive "System 1" decision model: give it a state (an email, a ticket, a JSON document) and a set of typed questions, and it returns typed decisions, a probability distribution over your options, and a calibrated confidence, all from a single forward pass of a 421M-parameter bidirectional encoder. No tokens are generated, so there is nothing to parse and nothing to hallucinate.

The timing is not a coincidence. Three days before Laya went public, TypeSafe AI opened early access to Jev, a closed-source model with the same shape of interface, and our earlier coverage of that launch is the natural prequel to this piece. Laya is, in every sense, the open answer to Jev: same three decision primitives, same RLCD training philosophy (reinforcement learning for calibrated decisions), Apache-2.0 weights you can host yourself, and a README that spends an unusual amount of space telling you where the model fails. This article walks through the full story: why generative LLMs are the wrong tool for reflex decisions, how Laya's architecture extracts decisions from [MASK] tokens, the strictly-proper-scoring-rule mathematics that makes its probabilities honest, what the benchmarks actually show, and the honest limits the promotional coverage leaves out.

One: the real problem, generative LLMs making reflex decisions

Every modern AI pipeline has the same bottleneck hiding in it: a generative large language model being used for decisions that are, structurally, trivial. A support ticket arrives and someone has to know which team owns it. An email lands and someone has to know whether it is phishing. A prompt hits an API and someone has to know whether it is a jailbreak attempt. A severity rubric says 0 to 3 and someone has to pick a level. Calling an 8B or 70B generative model for this is overkill in the literal sense: you wait 500 to 2,000 milliseconds for tokens to stream, you pay per-token inference cost, and then you write regexes or a JSON parser to extract a clean label out of free text that was never promised to be clean.

The deeper failure is epistemic. When an LLM prints confidence: 0.95, it is predicting a sequence of tokens that sounds confident; there is no mathematical calibration behind the digits. What these pipelines actually need is a System 1 model in the Kahneman sense: reflex-fast, honest about its uncertainty, and cheap enough to sit inside a hot loop. Laya's target is 30 to 40 milliseconds on commodity hardware with probabilities that were trained to be true, not typed to look true.

Two: three decision primitives, one forward pass

Laya accepts a state plus one or more typed questions and evaluates all of them in a single parallel forward pass. There are exactly three primitives, and the return shape defines each one:

• choice: pick one option from a dictionary of criteria; returns the selected label, a probability for every candidate, and a confidence score. Department routing, intent recognition, topic classification.
• score: place the state on an ordered rubric such as 0/1/2/3; returns an expected value (which can be fractional), per-level probabilities, and confidence. Customer frustration, urgency, harm severity.
• noul: answer a boolean question with a calibrated $P(\text{true}) \in [0, 1]$. Phishing, spam, jailbreak, churn risk.

Because the output space is restricted to probabilities and numbers, the model never generates text. It cannot hallucinate a label that is not in your option set, and it cannot emit malformed JSON, because it never emits text at all. The failure mode it does have, picking the wrong option among yours, is quieter, and we return to it below.

Three: the architecture, 421M parameters and a [MASK] trick

The lineage matters here. In March 2025 Nandakishor M published arXiv:2503.23303, a reinforcement-learning model that predicted sales-conversion outcomes turn by turn using frozen sequence embeddings plus a separate PPO value network. It worked, but it was not end-to-end and could not absorb new questions at runtime. A second paper in September 2025 (arXiv:2510.01237) laid out the schema-based decision framework. Laya is the reconstruction of that line of work into a general, horizontal System 1 engine, and the author's own framing of the motivation is blunt: he built this a year before a funded lab packaged the same idea as a product.

The ModernBERT-large backbone

The encoder is ModernBERT-large: 395M parameters, 28 layers, hidden dimension 1024, 16 attention heads, GeGLU intermediate layers of width 2624, and rotary position embeddings (RoPE) supporting 8,192 tokens. Because it is fully bidirectional, every token attends to the whole state and to every option simultaneously, which is exactly what a decision over a fixed option set wants and what a causal decoder cannot give you.

Option extraction at [MASK] positions

For each question, a build_sequence routine wraps the prompt so that every option owns exactly one [MASK] token:

After the encoder and a two-layer decision-head transformer (about 25.2M parameters, pre-LayerNorm, dropout 0.1), torch.gather pulls the hidden states at exactly those mask positions. An option-scorer MLP projects each 1024-dimensional state to a scalar logit, and a softmax over the options belonging to that question yields the candidate distribution:

$$p_k = \frac{\exp(z_k / T)}{\sum_j \exp(z_j / T)}$$

where $T$ is a temperature fitted per question type and option count. That temperature is not decoration: both shipped checkpoints are over-confident as released, and refitting one temperature per (question type, option count) on held-out data moves mean expected calibration error from 0.466 to 0.081 on the English checkpoint and from 0.314 to 0.106 on the multilingual one. The multilingual checkpoint ships with no fitted temperatures at all.

The act/escalate head

Real automation needs to know when not to trust itself. Laya adds a second head that takes the pooled [CLS] state (1024 dimensions) concatenated with four distribution features: the max probability $\max(p)$, the top-two gap $p_{(1)} - p_{(2)}$, the normalized entropy $H(p)/\log K$, and the option-budget ratio $K/255$. The resulting 1028-dimensional vector passes through a two-layer MLP producing $[P(\text{act}), P(\text{escalate})]$. As we will see in the limits section, this particular head is the one component the README tells you not to trust yet.

Many questions, one forward pass

Five questions about one email become five sequences packed into one batch; ModernBERT processes them in a single forward pass, about 35 ms on GPU. Batching across states is the same idea one level up: predict_batch packs many states against the same questions into shared forward passes, and on an RTX 5060 Ti per-decision latency drops from roughly 10 ms to about 1 ms. On a T4 the published numbers are 32.8 ms for one question and 7.2 ms per question at a batch of ten, with throughput of 103 to 332 questions per second.

Four: RLCD, the mathematics of honest probabilities

The interesting question is not how to make the model accurate but how to make its probabilities true. RLCD, reinforcement learning for calibrated decisions, answers it with decision theory.

Why cross-entropy and naive RL both break calibration

Train a classifier with cross-entropy,

$$\mathcal{L} = -\sum_k y_k \log q_k$$

and the loss is minimized only as the winning logit goes to infinity: the model is pushed toward certainty whether or not certainty is warranted. Naive reinforcement learning with a binary reward (+1 correct, 0 wrong) is no better. The expected reward is

$$\mathbb{E}[r] = \sum_k q_k \, y_k \cdot 1 = q_{y}$$

and the policy gradient drives the top probability to 1.0 and everything else to 0.0. Naive RL maximizes accuracy by destroying calibration; you get a machine that is confidently wrong.

Strictly proper scoring rules as reward

The fix is to make the reward a strictly proper scoring rule. A scoring rule $S(q, y)$ pays the model for reporting distribution $q$ when outcome $y$ happens, and it is strictly proper if and only if the expected score is uniquely maximized at $q = p$, the true distribution. Report honest probabilities, or lose reward: there is no other optimum. Laya's composite reward combines three such rules,

$$\begin{aligned} R(q, y) = {} & S_{\log}(q, y) \\ & + 0.5\, S_{\text{sph}}(q, y) \\ & - 1.0\, S_{\text{rps}}(q, y) \end{aligned}$$

each doing a different job.

The logarithmic score,

$$S_{\log}(q, y) = \sum_k y_k \log \max(q_k, 10^{-12})$$

punishes hard when the true outcome gets low probability; the floor at $-9.21$ (that is, $\log 10^{-4}$) keeps gradients finite. The spherical score,

$$S_{\text{sph}}(q, y) = \frac{\sum_k y_k q_k}{\sqrt{\sum_k q_k^2}}$$

is bounded in $[0, 1]$ and rewards putting mass on the right class without the extreme gradient spikes a pure log loss produces. The ranked probability score, used for ordinal score questions,

$$\begin{aligned} S_{\text{rps}}(q, y) &= \frac{1}{K-1} \sum_i \Delta_i^2 \\ \Delta_i &= \mathrm{CDF}_q(i) - \mathrm{CDF}_y(i) \end{aligned}$$

measures squared distance between cumulative distributions, so guessing level 2 when the truth is 3 is rewarded relative to guessing 0. Subtracting it teaches the model distance along the rubric, not just the argmax.

The policy gradient: REINFORCE with a group baseline

Training uses pure policy gradient with zero supervised cross-entropy. For each question the trainer samples $G = 8$ noisy candidate logit vectors,

$$z_{\text{noisy}} = z + \epsilon, \quad \epsilon \sim \mathcal{N}(0, \sigma^2)$$

projecting the noise so it sums to zero across options ($\epsilon - \bar{\epsilon}$), because adding a constant to all logits cancels in the softmax. The exploration width $\sigma$ anneals from 1.0 to 0.3. Each noisy vector defines a distribution $q_{\text{noisy}} = \mathrm{softmax}(z_{\text{noisy}})$ with reward $r = R(q_{\text{noisy}}, y)$, and advantages are taken against the group mean in GRPO style,

$$A = \frac{r - \mathrm{mean}(r)}{\mathrm{std}(r) + 10^{-6}}$$

with the policy loss built from the Gaussian log-probability of the sampled logits,

$$\mathcal{L}_{\text{policy}} = -\mathrm{mean}\left( A \cdot \log \pi(z_{\text{noisy}} \mid z) \right)$$

The cost-sensitive act head, and where 62.5% comes from

The act head samples from {act, escalate} against a cost matrix: act and correct $+1.0$, act and wrong $-3.0$, escalate $-0.5$. Writing $p_c = P(\text{correct})$ for the probability that an automated action is right, automated action is profitable only when

$$\begin{aligned} p_c (1.0) + (1 - p_c)(-3.0) &> -0.5 \\ 4\,p_c &> 2.5 \\ p_c &> 0.625 \end{aligned}$$

so the policy learns on its own to act only above 62.5% confidence. The threshold is not a hyperparameter you tune by feel; it is an arithmetic consequence of what a wrong automated action costs you relative to a human review. Change the cost matrix and the threshold moves with it.

Five: multi-turn trajectories with TD(lambda), and the leakage fix

The March 2025 paper predicted sales conversion across conversation turns with PPO, and surfaced a leakage bug worth remembering: feed the embedding of the whole conversation at turn 1 and the model reads its own future. Laya's fix is structural. Conversations are sliced into growing prefixes (turn 1, turns 1-2, turns 1-3, ...), the model only ever sees context up to the current turn, and training uses temporal-difference learning with Monte Carlo returns, TD($\lambda = 1.0$):

$$\begin{aligned} V(s_t) & \leftarrow V(s_t) + \alpha \left[ G_t - V(s_t) \right] \\ G_t & = \sum_{k \geq t} \gamma^{k-t} r_k \end{aligned}$$

With $\lambda = 1.0$ early turns are trained directly against the conversation's real terminal outcome instead of bootstrapping from the model's own guesses, so the model learns which early conversational patterns actually precede conversion or churn.

Six: confidence as normalized entropy

Every answer carries a confidence in $[0, 1]$ computed from the shape of its own distribution:

$$\begin{aligned} \text{confidence} & = 1 - \frac{H(p)}{\log K} \\ H(p) & = -\sum_k p_k \log p_k \end{aligned}$$

where $K$ is the number of options. Fully uniform, $p_k = 1/K$, gives entropy $\log K$ and confidence exactly 0.00; fully peaked gives entropy 0 and confidence 1.00. It is a dispersion statistic, not a learned head, which is why the selective-automation curve below is meaningful: accepting only the top 50% of predictions by confidence lifts accuracy from 83.8% to 92.2%.

Seven: the data pipeline, no synthetic shortcuts

The tempting way to build a model like this is to have an LLM generate synthetic questions and labels at scale. The author argues this is a trap for exactly the calibration reason above: a model trained on synthetic labels calibrates itself to the generating LLM's errors and hallucinations, not to reality. Laya's pipeline uses 100% human-annotated public datasets across support routing and intent, entailment and fact-checking, content-safety consensus labels, real jailbreak and prompt-injection attacks, human rubric ratings for helpfulness and correctness, and multi-turn SaaS sales and support conversations with verified outcomes. To block shortcut learning, the pipeline shuffles option order dynamically, paraphrases question phrasings, alternates between raw text and nested JSON states, and injects random distractor questions.

One caveat belongs here for honesty's sake: the 2025 SalesRLAgent abstract describes GPT-4o-generated synthetic conversations as part of that earlier dataset, while the "zero synthetic shortcuts" claim attaches to the current pipeline. The two statements are about different pipelines, but readers comparing papers should notice the difference rather than inherit the slogan.

Eight: benchmarks, and how to read them

The headline evaluation covers 25,424 questions: 23,024 in-task plus 2,400 zero-shot questions from task families never seen in training, compared against TypeSafe's published Jev figures.

Metric	TypeSafe Jev (published)	Laya (fine-tuned checkpoint)	Read
p50 latency, 1 question	~400 ms typical (70-500 ms; best 150 ms)	38.4 ms (p95 42.1 ms)	~10x on the average, ~4x on Jev's best case
10 questions batched	~1,500 ms serial / ~400 ms	156.0 ms (p95 158.4 ms)	Laya answers 10 in the time Jev answers 1
50 questions batched	seconds / rate-limited	721.4 ms	high-throughput small-batch regime
Workflow accuracy	67.8% over 4 production workflows	83.8% in-task macro average	+16.0 points, but see the caveats below
Intent and routing	~95-98% agreement	99.1% (ECE 0.009)	near-zero calibration error on routing
Moderation and safety	~92-95% agreement	96.7% (ECE 0.061)	clean safety boundaries
Selective automation, top 50% confidence	escalation claimed	92.2% (ECE 0.041)	the number that matters for gating
Weights and code	closed API	Apache-2.0, self-hostable	data sovereignty, air-gapped deployment
Inference cost	$0.042 per 1M input tokens	$0 self-hosted	GPU, Mac MPS, or CPU

The per-family table over the 23,024 in-task questions shows where the model is strong and where it is merely usable: intent and routing 99.1%, moderation 96.7%, topic classification 93.9%, sentiment 90.6%, entailment and fact-checking 88.3%, instruction-following 87.8% in-task and 86.3% zero-shot, robustness checks 85.1%, reading comprehension 84.7%, email triage and phishing 73.2%, search relevance 62.8%, and reply-quality scoring 58.1%. The zero-shot macro average over 2,400 held-out questions is 65.1%, dragged down by sentiment (58.3%) and sentiment-rating (36.2%) families.

Two independent measurements keep the latency claim honest from the other side. Third-party Jev benchmark repositories (AbdelStark/jev-benchmarks and nibzard/decision-model-benchmark) put Jev's p50 at 236 to 276 ms, not the 150 ms best case the marketing quotes; against that, Laya's 32.8 ms routed single-question latency is about 7.8x. And the shared-benchmark table in the README, where every Laya figure is what Router().predict(...) actually returns, gives Laya 0.766 versus Jev's published 0.727 on typed-decisions, above the 0.735 teacher self-agreement ceiling, with 2.4x better Brier score.

Nine: what the promotional article does not say

The WeChat post and the dev.to essay are launch pieces; the README's "Honest limits" section is the engineering document, and it changes how you should deploy this model. Five items matter most.

The base checkpoints are near chance zero-shot. On typed-decisions the un-fine-tuned checkpoints score 0.362 and 0.342 against a 0.318 random baseline and a 0.461 majority-class baseline. The celebrated 0.766 comes from the checkpoint fine-tuned on that benchmark's own training split. Laya is a fast base to specialise, not a zero-shot decision engine. Anyone quoting 0.766 without the fine-tuning clause is misreading the table.

Khmer scores 0.000 accuracy at 95.2% confidence. The English checkpoint collapses on non-Latin scripts while remaining maximally confident, which is precisely the case where confidence gating cannot save you: the gate sees 0.952 and lets garbage through. This is why the Router exists, detecting script in under 0.5 ms of pure Python before the forward pass and dispatching to the multilingual checkpoint. Across all 51 languages tested, the English checkpoint macro-averages 0.227 with macro ECE 0.733, and only 23 of 51 languages clear 3x random; routed, 45 of 51 do.

High-cardinality choice questions hit a token budget. Options share a fixed head_max_len budget (192 tokens on the English checkpoint, 256 on multilingual). On Banking77's 77 labels that leaves about 3 to 4 tokens per label and accuracy falls to 0.425, while Jev scores 0.870 on 72 labels with support for up to 255 options out of the box. The workarounds are raising head_max_len at runtime, embedding-based shortlisting via predict_shortlist, or splitting the label set into coarse and fine questions.

noul can follow its labels instead of the state. The default rendering uses false: / true: as the two option strings, and on the English checkpoint that label pair can dominate the answer, returning a confident "no" for clearly positive input (issue #156). The multilingual checkpoint has a related position bias on score questions, rarely choosing the first-listed level (#131). Both have documented workarounds, overriding the model-facing labels or using a two-option choice with neutral keys, and both come with the instruction to validate on your own data.

act_probability carries no usable signal yet. It reads 1.0 for almost every input and its raw logits run against correctness (AUROC 0.30 on 396 labelled decisions). Gate on confidence instead, which reaches AUROC 0.77 on the same items. The act/escalate head described in the architecture section is real, but it is not the gate you should wire into production today.

Two smaller asymmetries round out the picture: Jev matches the teacher's full distributions better (soft accuracy 0.580 versus Laya's 0.471), and Laya's raw out-of-the-box ECE (0.213) is worse than Jev's published 0.144 until temperature fitting brings it to 0.081. Calibration here is a fitted artifact, shipped with the checkpoint, not a free property of the architecture.

Ten: running it, and the Jev-compatible surface

The package is a one-liner, pip install laya (v0.3.8 at time of writing), with weights on Hugging Face and ModelScope. The recommended entry point is the router:

from laya import Router

router = Router(preload=True)   # all checkpoints resident; language flips cost <1 ms

state = {
    "from": "user@company.com",
    "subject": "Charged twice on March invoice",
    "body": "Hi, we were billed twice for invoice 4411. Please refund the duplicate today.",
}
questions = {
    "department": {
        "type": "choice",
        "instructions": "Which department should handle this email?",
        "criteria": {
            "billing": "invoices, payments, refunds",
            "technical": "bugs, outages, system errors",
            "sales": "pricing, new contracts",
            "other": "everything else",
        },
    },
    "urgency": {
        "type": "score",
        "instructions": "How urgent is this request?",
        "criteria": ["not urgent", "soon", "critical deadline or blocking issue"],
    },
    "churn_risk": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?"},
    "is_phishing": {"type": "noul", "instructions": "Is this email a phishing or scam attempt?"},
}

result = router.predict(state, questions)   # one forward pass, ~33 ms on GPU
a = result["answers"]
print(a["department"]["choice"], a["department"]["confidence"])  # billing, 0.94
print(a["urgency"]["score"])                                     # 1.84 / 2.0
print(a["churn_risk"]["noul"])                                   # 0.892
print(a["is_phishing"]["noul"])                                  # 0.008

Because the probabilities are trained to be calibrated, the gating logic stays boring:

dept = a["department"]["choice"]
conf = a["department"]["confidence"]

if conf >= 0.85:
    route_automatically(dept)
else:
    send_to_human_triage(dept, reason=f"Low confidence ({conf:.2f})")

At a 0.85 gate the published curve says you automate roughly half your traffic at 92.2% precision and escalate the genuinely hard half. Deployment options span a Jev-compatible HTTP server (laya-serve speaks POST /v1/systemone, so a TypeSafe client migrates by changing the base URL), an MCP server, LangChain/LangGraph routing and guardrail nodes, a TypeScript package, ONNX and TileLang fast paths, and a free hosted endpoint at impossibl that is wire-compatible with TypeSafe's System One API. The fine-tuning notebook runs the whole loop, dataset build, RLCD training, temperature fitting, evaluation, Hub push, on Kaggle's free 2xT4 GPUs in four to five hours for four epochs over roughly 30k questions.

Eleven: what this means for agent and robotics harnesses

Our coverage of Jev ended with the observation that System 1 models compete for the slots in an agent harness where calling a frontier model every time is too slow or too expensive: should this trajectory get human review, is this tool result compliant, does the perceived state satisfy a precondition, which discrete action applies to this frame. Laya puts that slot within reach of teams that cannot send their data to a closed API at all, which in robotics and industrial settings is often the binding constraint rather than cost.

The strongest evidence that the loop closes is a documented specialisation rather than a demo: Laya as the operation-and-target decision head inside browser-use/jev-ultrafast, the same request format as Jev. Element top-1 among roughly 45 candidates goes from 0.10 zero-shot to 0.66 after fine-tuning on a single 16 GB GPU, and real-task success goes from 0% to 62% at 17 to 23 ms per step. Read that as the template: a 421M bidirectional encoder, specialised on your own states and your own option sets, sitting inside a control loop that a VLM could never keep up with. Community ports to Huawei Ascend NPUs report 34x to 71x speedups over CPU at batch size 1, which is the edge-deployment direction this class of model makes plausible.

The honest reading is the same one the README gives: treat Laya as a fast base to specialise. The zero-shot numbers are near chance, the multilingual story needs the router, high-cardinality label sets need shortlisting, and the act head is not yet a gate. What you get in exchange is Apache-2.0 weights, a training recipe you can reproduce on free GPUs, and probabilities whose calibration is a measured, fitted, published number rather than a marketing adjective. For a control loop, that is the difference between a component you can certify and one you can only hope about.

Pre-deployment checklist

1. Fine-tune on your own states and option sets; do not ship a zero-shot base checkpoint into a decision path.
2. Fit temperatures per (question type, option count) on held-out data, and re-fit after every retrain; stale bucket temperatures silently mask a new fit.
3. Route by script before the forward pass; never let confidence gating be your only language defence, because the confident failure mode is exactly Khmer-at-95.2%.
4. Avoid boolean-word labels in choice and check noul label sensitivity on your checkpoint; use semantic or opaque keys and validate.
5. Keep option sets under about 20 at default budgets, or raise head_max_len, shortlist with embeddings, or split coarse/fine.
6. Gate automation on confidence, not act_probability; derive your threshold from your own cost matrix the way the 62.5% figure is derived.
7. Monitor the silent failure class, wrong option among yours: typed outputs mean no parser will ever alert for you.

Compiled from the ModelScope community WeChat post (2026-09-21), the author's dev.to engineering write-up, the GitHub repository and its BENCHMARKS.md, and the Hugging Face model cards; star counts and checkpoint metadata verified on 2026-09-23. Promotional boilerplate in the source post (follow prompts, donation buttons) has been removed.

• Source article (WeChat, ModelScope community): Laya open-sourced: 4x faster than Jev, 421M parameters, 33 ms System 1 decisions
• Author's engineering write-up: dev.to: I built non-autoregressive decision models a year ago...
• Code: NandhaKishorM/laya (Apache-2.0)
• Weights: convaiinnovations/laya, laya-multilingual, laya-typed-decisions; ModelScope mirror convaiinnovations/laya
• Prior papers: arXiv:2503.23303, arXiv:2510.01237
• Related coverage on this site: TypeSafe Jev and the System One Model category, building your own Jev-class decision model