Laya: A Single-Pass Typed Decision Engine

Decision-making separated from long-form generation

Laya is a non-autoregressive decision engine. It does not continue a passage and leave the caller to infer an answer from prose. It reads a state, evaluates typed questions, and returns a selected option, an ordinal value, or a probability. That difference is more than an interface preference. Without generated text there is no output-parser failure. Without sampling, several questions can share one forward pass. A choice answer can expose its full distribution instead of collapsing to an unexplained label.

The repository is at version 0.4.13, licensed under Apache-2.0, and requires Python 3.10 or later. Its clearest role is the “system one” layer of an agent workflow: routing a request, selecting an action from a short list, classifying a ticket, deciding whether a case should escalate, or assigning an ordered severity level. It is not a replacement for the language model that handles open-ended reasoning and writing. It is a way to remove frequent, schema-bound, latency-sensitive judgments from that model.

The central trade-off is explicit. A bidirectional encoder with roughly 322M to 421M parameters performs the decision locally. An upstream agent or larger language model remains responsible for open-ended work. The useful question is therefore not whether Laya is generally smarter than a chat model. The question is whether a particular decision benefits from being isolated as a calibrated, batchable component with a stable output shape.

flowchart LR
    A["Text, email, ticket, or JSON"] --> B["Router detects script and language"]
    B --> C["English / Multilingual / Typed-decisions"]
    C --> D["Bidirectional encoder"]
    Q["choice / score / noul questions"] --> D
    D --> E["Typed decision heads"]
    E --> F["Label and full distribution"]
    E --> G["Ordinal score"]
    E --> H["P(true)"]
    F --> I["Business policy and thresholds"]
    G --> I
    H --> I

Three checkpoints and a request-level router

Three pretrained checkpoints share the typed interface but serve different traffic. The English and typed-decisions checkpoints use ModernBERT-large and have about 421M parameters each. The multilingual checkpoint uses mmBERT-base and has about 322M parameters. The router keeps at most two checkpoints resident by default to limit memory. Preloading removes cold-switch costs but raises the resident-memory floor.

Checkpoint	Encoder	Parameters	Context	Intended use
english	ModernBERT-large	about 421M	512 tokens	English classification, routing, ranking, and decisions
multilingual	mmBERT-base	about 322M	1,024 by default; configurable to 8,192	Non-English text and cross-language tickets
typed-decisions	ModernBERT-large	about 421M	1,024 tokens	Four fine-tuned decision workflows

The router examines script and language before inference. An explicit model wins first. The remaining precedence includes task selection, optional workflow detection, language hints, detected language, and the default checkpoint. That order matters in production. A long English document belongs on the English checkpoint, while mixed-language traffic may need the multilingual model pinned explicitly. Services should log the routed model on every request. Otherwise a change in accuracy cannot be separated from a change in input language distribution.

typed-decisions is never a silent default. It was fine-tuned on four exact question-identifier workflows, and automatic task detection requires an exact set match. A generic request that happens to contain a field named “risk” or “urgency” will not be mistaken for a trained workflow. The checkpoint is reachable only through an explicit task or model choice, or through the opt-in auto-detection mode. This is conservative, and it keeps a narrow fine-tune from capturing unrelated traffic.

The three question types represent three statistical objects

The question type determines the output shape. Encoding a yes/no question as two ordinary choices, or treating ordered levels as unordered categories, throws away information the caller needs. Laya keeps the three objects separate:

Type	Input	Primary output	Correct use
choice	Labels with optional descriptions	One label and the full distribution	Ticket categories, intent routing, action selection
score	Levels in ascending order	Expected ordinal value, legend, and full distribution	Severity, satisfaction, graded risk
noul	A yes/no instruction and optional labels	Probability that the proposition is true	Escalation, churn, rule firing

A choice question is positional. Labels return exactly as supplied, and descriptions are worth writing because the rendered descriptions are part of what the model reads. Two bare labels with no distinction give the model little evidence. Saving only the argmax discards the distribution that can support thresholds, abstention, and downstream policy.

The score field is an expected value over the ordered distribution, not the index of the most likely level. A five-level answer can return 2.90 while the highest probability sits on level 3. The expectation minimizes squared error across the scale; the argmax minimizes disagreement with the model’s preferred label. They answer different questions. The interface returns a legend so callers can interpret indices without guessing what each number means.

noul returns P(true), not a boolean chosen by the model. The threshold is business policy because false positives and false negatives have different costs. Callers can relabel the two sides, but labels affect the result. The English checkpoint is particularly sensitive to wording, so any relabeling must be evaluated on held-out data. “Automatic” and “escalate” are not merely presentation strings.

One state, several typed questions

A request can combine a choice, an ordinal score, and a yes/no decision around one state. The point is not syntactic convenience. A support ticket should be read once and yield a department, an escalation decision, and a severity estimate.

import laya

agent = laya.load("convaiinnovations/laya")
questions = {
    "department": {
        "type": "choice",
        "instructions": "Which team should handle this?",
        "criteria": {
            "billing": "charges, refunds, and invoices",
            "technical": "bugs, outages, and technical support",
            "sales": "pricing, contracts, and procurement"
        }
    },
    "urgent": {
        "type": "noul",
        "instructions": "Should this be escalated immediately?"
    },
    "severity": {
        "type": "score",
        "instructions": "How severe is the problem?",
        "criteria": ["minor", "low", "moderate", "serious", "critical"]
    }
}

result = agent.system_one(
    {"text": "I was charged twice, nobody replied for a week, and I need a refund now."},
    questions
)

department = result["answers"]["department"]["choice"]
urgent_probability = result["answers"]["urgent"]["noul"]
severity = result["answers"]["severity"]["score"]

The example also marks the boundary. Laya supplies probabilities. The external state machine still decides whether a refund is allowed, whether an incident is escalated, and which tools may run. Treating a model probability as authority confuses calibration with authorization.

Batching, long documents, and token budgets

Instructions, descriptions, and candidates consume the question head. A larger head_max_len leaves less room for the state, and more candidates leave fewer tokens per candidate. This is why a single choice question does not scale to hundreds of labels. At 77 labels, each label receives so little space that the options stop being distinguishable. The repository recommends keeping choices under roughly 20 options. Larger label sets should be recalled or shortlisted before the final decision.

The multilingual checkpoint has a default context of 1,024 tokens and can be raised to 8,192. Short inputs do not become slower merely because the limit is larger; inference follows the real input length. Long inputs need attention to both state room and head budget. Truncation should be treated as observable behavior: record what was dropped and where, rather than assuming the model saw the whole state.

When a document is longer than one sequence, predict_long scans overlapping windows and aggregates their answers. A yes/no question takes the strongest window, while choice and score answers take the most confident window. This can locate local evidence across a long document, but the returned probability belongs to the deciding window. It is not a jointly calibrated probability for the entire document. High-stakes automation must preserve that distinction.

Batching is where the throughput advantage becomes operational. Multiple questions share a forward pass, and multiple requests can be grouped into a batch. Mixing short and long states in one pass wastes compute on padding. Length sorting groups comparable sizes and reduces that waste. Production metrics should include batch size, questions per pass, real token length, timeout, and routed model; a warm single-request benchmark is not enough.

Structured projection and integration

The native answer is already structured. No generation means no JSON syntax error and no “missing quote” parser failure. It does not mean the semantic decision is correct. Label meanings, question wording, candidate descriptions, and training distributions can still produce a wrong answer. Tests should separate parse safety, which can be deterministic, from semantic quality, which requires held-out data and domain acceptance.

• The HTTP service exposes single and batch endpoints with controls for lang, max_len, head_max_len, and minimum-confidence behavior.
• Clients cover Python, TypeScript, Go, Java, Rust, and .NET; the TypeScript client can call a self-hosted service.
• Adapters connect the output to MCP, LangChain, LangGraph, LlamaIndex, and CrewAI.
• ONNX Runtime supports deployment-oriented inference; local PyTorch is the path for experimentation and retraining.
• Structured extras project answers into JSON Schema or Pydantic models, but application code should still validate enums, ranges, and required fields.

Laya does not need to replace a model gateway. A practical deployment places it behind the gateway as a specialist: stable schema on the left, distributions on the right, and an outer policy that decides whether to call a larger model, invoke a tool, or ask a human. That separation lets routing and thresholds evolve without redesigning the rest of the generation stack.

Training and calibration focus on distribution quality

The project calls its training method RLCD. Alongside supervised learning, it adds a reward based on strictly proper scoring rules and combines that signal with soft cross-entropy. A strictly proper scoring rule rewards the model for reporting its true belief distribution, which matters when the product is the full distribution rather than only an argmax. The reinforcement signal is not free-form text generation; it is aimed at the quality of the probabilistic decision.

For deployment, the important step is not to sharpen confidence until the numbers look impressive. Temperatures should be fitted by question type and option count on held-out data, then checked on a separate validation split. The repository reports expected calibration error falling from 0.466 to 0.081 for the English checkpoint and from 0.314 to 0.106 for the multilingual checkpoint. The improvement shows that calibration is actionable. It also shows that shipped weights should not be treated as production-calibrated probabilities.

The checkpoints ship overconfident, and the multilingual checkpoint has no fitted temperatures at all. A threshold measured on three-option questions does not transfer unchanged to twenty-option questions. Callers should also keep answer_confidence and entropy-based confidence distinct. The first is the quantity temperature scaling fits. The second describes how concentrated the whole distribution is. They live on different scales and must not share a threshold.

How to read the evidence

The repository includes reproducible experiments, raw result files, and charts. The figures below use only settings that can be tied to a concrete checkpoint and dataset. They do not merge controlled experiments, third-party published results, and different hardware into a single universal score.

On the multilingual T4 measurement, one question has a 32.8 ms p50. A ten-question batch takes 72.3 ms total, or 7.2 ms per question. The router raises the number of MASSIVE languages above three times random from 23 of 51 to 48 of 51. Routing clearly helps, but the remaining difficult languages do not disappear.

Evidence	Value	Defensible reading
Multilingual one-question latency	32.8 ms p50	A dedicated T4 decision layer is fast
Ten-question batch	72.3 ms total, about 7.2 ms each	Shared forward passes improve throughput
Languages cleared by the router	23/51 to 48/51	Routing materially improves language coverage
Fine-tuned typed-decisions result	0.766 accuracy	Measured on the checkpoint trained for that benchmark
English calibration	ECE 0.466 to 0.081	Refitting on held-out data materially helps
Requests with about 4,000 preceding tokens	16 to 18 of 20 correct	Useful signal, but validate on the target documents

The TypeSafe Jev comparison needs restraint

The repository identifies Jev’s 0.727 value as a third-party published result, not a controlled rerun on identical hardware, data order, and training budget. Laya’s 0.766 comes from a checkpoint fine-tuned on the corresponding training split. The numbers suggest competitiveness on some tasks. They do not support a claim that Laya is universally better than Jev. A procurement comparison should rerun both systems with fixed data, prompts, option order, hardware, and timeouts.

Limits that belong before the adoption decision

One of the project’s strengths is that its own documentation exposes failure cases. The following constraints determine whether it belongs in production; they are not optional footnotes.

• Zero-shot typed decisions are close to chance. The base checkpoint scores about 0.36 on the relevant test, against 0.318 random and 0.461 for the majority class. The 0.766 result belongs to a checkpoint fine-tuned on that benchmark’s training split and does not represent out-of-the-box accuracy.
• More candidates make discrimination harder. Keep choices under roughly 20 options. At 77 labels, the fixed head budget leaves too few tokens per label.
• Negation is unreliable. A state saying not to cancel an account can still be classified as cancellation with high confidence, especially when labels are semantically close.
• Relabeling yes/no answers introduces label bias. The model can follow a label pair instead of the state. Any relabeling needs a fresh evaluation.
• Long-document probability is window probability. The deciding window’s confidence is not a calibrated confidence for the full document.
• Moderation is weak on held-out data. Toxic-chat accuracy is about 0.530 with 0.400 macro-F1. Curated demo examples are not evidence about real traffic.
• The action head is not dependable. The repository reports action probability near 1 almost everywhere and an AUROC around 0.30. It cannot replace an authorization system.
• Multilingual scoring has a position bias. Ordered scales should be checked separately by language and level count.

“No generation, therefore no hallucination” is the wrong conclusion. The precise claim is that Laya does not generate a passage that needs parsing, so generated-text syntax failures do not apply. It can still make the wrong semantic decision because labels, instructions, or training data point it in the wrong direction. Reliability comes from data, calibration, monitoring, and fallback policy, not merely from returning JSON-shaped output.

A practical adoption path

The safest first deployment is an observable assistant rather than an autonomous final decision. Replay historical tickets offline, measure each supported language, fit temperatures per candidate count, and introduce low-risk actions gradually. At minimum, the rollout should include the following checks.

1. Create an independent held-out set stratified by language, text length, option count, and class, with no overlap with training or tuning data.
2. Store the full distribution rather than only the argmax, together with routed model, actual token count, truncation, threshold, and abstention state.
3. Keep answer_confidence, entropy-based confidence, and business confidence under separate names so they cannot be assigned the same threshold by accident.
4. Add regressions for negation, double negation, semantically similar labels, long candidate lists, and low-resource languages.
5. Maintain a fallback to a larger model or human review, with separate cost estimates for false positives and false negatives.
6. For long-document scans, record which window supplied the answer and add a conflict policy instead of treating one window’s probability as the document probability.

Local operation requires Python 3.10 or later. HTTP, MCP, ONNX, and the various language clients are optional installs. The first checkpoint download needs access to Hugging Face, while offline deployments should cache the weights in advance and pin a revision. CPU-only operation is supported, but latency and concurrency limits must be measured on the target machine. Loading several checkpoints at once raises memory use, and the default two-checkpoint residency cap exists to prevent a host from quietly loading everything.

The durable value of Laya is not a first-month leaderboard result. It is a clear engineering pattern: isolate common decisions into a specialist model, retain full probability distributions for external policy, and make training and calibration reproducible. For teams with a high volume of structured decisions that do not need a generative model on every step, one forward pass, a typed output, and language-aware routing are closer to an operable system component than a chat interface. That remains true only if the documented limits are treated as part of the deployment, with every decision boundary revalidated on local data.

Repository: NandhaKishorM/laya.