PAPER DEEP DIVE
Rufus-Air: An Open LLM Post-Training Recipe
Rufus-Air is an open and reproducible post-training recipe on GLM-4.5-Air-Base (106B-A12B), organized as a serial pipeline of eight stages: SFT, Reasoning RL, Coding RL, Instruction-Following RL, General Agent, Coding Agent, Search Agent, and RLHF. We document the data, reward design, infrastructure, stage order, and stagewise results needed to reproduce the recipe. Stages progress from basic to advanced capabilities and from hard, verifiable rewards to softer judge-based signals. Training builds on open-source components and public data, much of it used as released, without new human annotation or an in-house distillation teacher. Our main findings are that (i) diverse, high-quality SFT establishes a strong capability floor; (ii) difficulty filtering keeps RL prompts within a productive learning range; (iii) reward reliability provides a practical principle for ordering stages; and (iv) infrastructure and engineering choices are part of the recipe, not just an implementation detail. Rufus-Air improves over the official GLM-4.5-Air post-trained release and is competitive with similarly sized open models.
One-Sentence Summary
Rufus-Air turns post-training into an eight-stage engineering system: a broad SFT mixture establishes the capability floor, difficulty filtering creates a live curriculum, reward reliability determines stage order, and token-faithful sandboxed rollouts make long-horizon agent training practical.
Why This Paper Exists
Open-weight checkpoints have made model adaptation widely accessible, but useful post-training recipes remain unusually scarce. A team can obtain the initial weights and still lack the information needed to reproduce a strong endpoint: the data mixture, the filters that determine which prompts generate useful gradients, the ordering of several RL stages, and the path by which multi-turn tool calls become training tokens.
The authors use GLM-4.5-Air-Base as a fixed starting point. The model has 106B total parameters and 12B active parameters in a mixture-of-experts architecture. Its public weights, tokenizer, and native thinking format fit an open stack built from Slime, Megatron-LM, and SGLang. The size is also deliberate: it is large enough to expose real MoE routing issues, while RL stages still fit on 8 to 32 nodes in the reported experiments.
Rufus-Air is a serial rather than ensemble pipeline. SFT is followed by Reasoning RL, Coding RL, Instruction-Following RL, General Agent, Coding Agent, Search Agent, and finally RLHF. Every stage trains from the checkpoint produced by the preceding stage. This makes transfer visible, but it also means later updates can erase earlier gains. The paper therefore measures each stage against its own starting checkpoint whenever possible.
The ordering principle is the most reusable contribution. Stages move from basic capabilities toward specialized agent behavior, and from deterministic rewards toward noisy learned preferences. Hard verifiers run early. Open-ended reward models run last because their weaknesses have more ways to be exploited and should remain under optimization pressure for less time.
The Full Pipeline
Figure 1: The recipe begins with GLM-4.5-Air-Base and applies eight successive training stages, each targeting a distinct capability and reward regime.
flowchart LR B["Base model
106B total / 12B active"] S["SFT
broad capability floor"] R["Reasoning RL
deterministic verifier"] C["Coding RL
execution tests"] I["Instruction RL
necessary rubrics"] G["General Agent
MCP state transitions"] A["Coding Agent
shell sandbox"] W["Search Agent
noisy judge"] H["RLHF
learned preference"] F["Rufus-Air
final checkpoint"] B --> S --> R --> C --> I --> G --> A --> W --> H --> F
If each checkpoint is represented by a policy $\pi_k$, the recipe is:
$$\pi_0 \xrightarrow{\mathrm{SFT}} \pi_1 \xrightarrow{\mathrm{Reasoning\ RL}} \pi_2 \xrightarrow{\mathrm{Coding\ RL}} \pi_3 \xrightarrow{\mathrm{IF\ RL}} \pi_4 \xrightarrow{\mathrm{General\ Agent}} \pi_5 \xrightarrow{\mathrm{Coding\ Agent}} \pi_6 \xrightarrow{\mathrm{Search\ Agent}} \pi_7 \xrightarrow{\mathrm{RLHF}} \pi_{\mathrm{Rufus}}$$The diagram is not merely descriptive. It defines how the paper attributes progress. Adjacent checkpoints under the same evaluation protocol carry the stage-level claim; a comparison between the base model and the final model describes the combined effect of the entire recipe.
Stage One: SFT Sets a High Floor
The SFT corpus contains 9.01 million samples, 44.5 billion raw tokens, and 66.7 million conversational turns. Of those turns, 23.5 million are supervised. After masking system messages, user turns, and tool observations, 27.04 billion tokens contribute to the loss. This is far more than a formatting warm-up: the mixture introduces mathematical reasoning, code generation, long-context dialogue, and tool-use trajectories.
Sample count and supervision volume tell different stories. General Agent data represents 39.6% of samples but only 14.2% of training tokens. Math and Coding Agent together represent 18.8% of samples yet supply 49.1% of supervised tokens because their reasoning traces and trajectories are much longer. Reporting both views prevents source diversity from being mistaken for equivalent training signal.
Figure 2: The largest category by sample count is not the largest source of loss-bearing tokens.
Preprocessing normalizes role-aware conversations and loss masks. In agent trajectories, a thought-action-observation chain remains supervised only when a tool result follows the assistant action. Earlier dialogue stays as context, but earlier rationales are masked. The final user request then anchors the response that the model is trained to produce. This prevents historical thinking from being treated as part of the current answer.
Decontamination combines sampled 8-gram overlap against reported benchmarks with exhaustive n-gram and embedding retrieval for math and knowledge sets. The first ten benchmark passes remove 3,529 samples. The stricter math and knowledge review identifies another 658 unique samples, bringing the total to 4,187 removed instances. The authors explicitly note that no such filter can prove the absence of leakage, so the same screening is reused when RL prompt sets are assembled.
SFT runs for three epochs on 512 H200 GPUs with a batch size of 4,096, a $5\times 10^{-5}$ peak learning rate, and cosine decay to $5\times 10^{-6}$. Training loss falls from 0.84 to 0.34, but held-out evaluation flattens within the first epoch. The team ships checkpoint 3,799 instead of the final training step: lower loss on repeated data is not evidence of better generalization.
Figure 3: Loss steps down at epoch boundaries while validation curves remain near a plateau; the RL pipeline starts from checkpoint 3,799.
| Checkpoint | IFEval | IFBench | GPQA | AIME 25 | AIME 26 |
|---|---|---|---|---|---|
| Public GLM-4.5-Air | 83.00 | 33.60 | 73.90 | 84.20 | 86.50 |
| Rufus-Air SFT 3799 | 88.33 | 57.75 | 68.18 | 90.83 | 90.00 |
| Difference | +5.33 | +24.15 | -5.72 | +6.63 | +3.50 |
The SFT model already beats the public post-trained release on single-turn instruction following and both AIME sets, while trailing on GPQA. That mixed profile is useful: it shows that SFT establishes a strong but uneven floor, and it leaves identifiable targets for the RL stages.
Reasoning RL: Data Selection Does Most of the Work
The reasoning stage uses 121,161 prompts spanning math, science, and algorithmic puzzles. Math accounts for 47.7% of prompts, while puzzles account for only 16.7% of prompts but 55.5% of prompt tokens because their boards, grids, and rule descriptions are long. Math uses canonical answer matching, science uses fuzzy string matching, and puzzles use generated Python checkers.
A teacher-based correctness filter removes prompts for which GPT-OSS-120B cannot obtain a positive reward. A learnability filter then removes prompts the current policy solves more than 80% of the time and prompts with no observed success. During training, the system oversamples candidates and retains groups whose mean reward remains inside a productive band.
$$\bar r=\frac{1}{G}\sum_{j=1}^{G}r_j,\qquad \bar r\in(0.0,0.8]$$Here $r_j$ is the reward of rollout $j$ and $G$ is the group size. Because the policy changes during training, previously impossible prompts can enter this band later. The filter behaves as an automatic curriculum, replacing hand-authored difficulty tiers.
The optimizer is sequence-level GSPO, chosen for stable long MoE rollouts rather than as a claim of a new objective. Truncated responses are masked, and a linear length penalty discourages answers longer than the shortest correct rollout. Rollout Routing Replay preserves expert routing between sampling and the BF16 training forward pass.
Against the SFT checkpoint, GPQA rises from 68.18 to 73.50. AIME 25 and AIME 26 fall by 2.81 and 2.60 points. The authors treat AIME as a guardrail instead of the optimization target: each set has only 30 questions, and repeated runs of AIME 25 at $n=8$ can differ by 4.6 points. The defensible conclusion is therefore a clear GPQA gain with AIME remaining within evaluation noise, not evidence of a broad reasoning collapse.
Coding RL: The Verifier Is the Environment
The coding set contains 29,405 Python problems after filtering. It mixes functional tasks validated by assertions with stdin/stdout problems validated by exact input-output behavior. Sources include EvolveCoder, Nemotron, Dolci, and ADR. Records without tests are removed, statements are deduplicated, a teacher must solve each retained item at least once, and problems solved by all four warm-up samples are dropped.
The parser extracts the last fenced Python block and executes it in a sandbox. Up to 50 tests are selected with a deterministic hash-based seed, so every rollout of a problem sees the same subset. A completion receives reward 1 only if every selected test passes:
$$R_i=\mathbf{1}\left[\text{all selected verifier tests pass}\right]$$No format bonus is needed because 98% to 100% of initial responses already use the required fence. The reward therefore rewards executable correctness rather than the presence of code-shaped text.
The most consequential intervention was not a change to the policy loss but an increase in the response budget. For the first 23 rollout steps, 11% to 28% of completions reached the 64K token cap and were removed from the loss. Extending the limit to 128K reduced truncation below 0.1%. Training reward then continued to 0.42, and LiveCodeBench v6 pass@1 reached 75.9 at step 33.
Figure 4: The 64K phase supplies most of the gain; the 128K extension removes truncation and supports a later pass@1 peak of 75.9.
| Adjacent checkpoints | LiveCodeBench pass@1 | LiveCodeBench pass@8 |
|---|---|---|
| Reasoning RL | 68.6 | 85.1 |
| + Coding RL | 75.9 | 87.4 |
| Stage gain | +7.3 | +2.3 |
Because pass@1 improves much more than pass@8, the stage mostly teaches the policy to solve a problem on its first attempt. The paper explicitly warns that this stage-specific protocol is not directly comparable with the headline LiveCodeBench number in the final results table.
Instruction RL: Reward Necessary Constraints Only
Instruction-Following RL combines 14K synthesized single-turn prompts with 13K multi-turn conversations. Single-turn examples contain 2 to 6 atomic constraints, each backed by a Python checker. Multi-turn examples average 6.1 turns and are generated adversarially: a teacher plays the user while several open models rotate as assistants. The user side actively adds constraints, changes prior requirements, and inserts distractors.
Every training example has a list of necessary rubrics. Mechanical conditions use Python functions; nuanced conditions use an LLM judge. Positive reward requires all rubrics to pass:
$$R_i=\mathbf{1}\left[\bigwedge_{j=1}^{m_i}c_{ij}=1\right]$$Here $m_i$ is the number of required conditions for example $i$, and $c_{ij}$ indicates whether condition $j$ is satisfied. The key design choice is the exclusion of optional rubrics. In a negative experiment, adding about ten desirable-but-not-required rubrics caused a pronounced length bias because the policy tried to cover every possible credit.
With necessary-only scoring, final-answer length falls from about 5.4K to 3.6K tokens while all four instruction benchmarks improve. The model learns that extra prose can violate a constraint rather than demonstrate compliance.
Figure 5: Training reward rises while final-answer length falls, consistent with more precise rather than more verbose responses.
IFEval moves from 90.5 to 94.5, IFBench from 63.8 to 77.8, Multi-challenge from 31.1 to 55.8, and AdvancedIF from 45.5 to 62.9. GPQA rises 3.2 points and AIME 25 falls only 1.2. The authors also report that applying this stage after Reasoning RL produces larger gains, especially on AdvancedIF, supporting the sequence in which reasoning acts as a prerequisite for complex constraint satisfaction.
General Agent: Optimize the Final Environment State
General Agent training starts from AgentWorldModel, which provides 10K single-server MCP tasks across 1K synthetic environments. Each task has a database schema and roughly 35 tools. Tool calls mutate server state, and a rule-based verifier checks the terminal state. A correctness pass ensures that a teacher or the initial policy has solved each retained task at least once; difficulty filtering then removes groups with no useful reward variation, leaving about 2K tasks.
$$R_i=\mathbf{1}\left[\mathrm{poststate}_i\models\mathcal{C}_i\right]$$$\mathrm{poststate}_i$ is the server state after the episode and $\mathcal{C}_i$ is the task condition. There is no step-level shaping or LLM judge. The agent receives a binary signal only after the interaction ends, which makes the environment semantics and verifier reliability central parts of the algorithm.
GRPO with DAPO-style dynamic sampling removes all-pass and all-fail groups. Each training step runs 3,072 trajectories, and MCP server I/O, database mutation, transport, and verification are CPU-heavy. Rufus-Gym distributes environment actors with Ray so that tool execution does not starve the GPU training process.
Figure 6: General-agent reward and held-out score rise together while the median trajectory grows from about 14K to 16.5K tokens.
From the IF RL checkpoint, MCP-Atlas rises from 34.95 to 42.75 and Tau2-Retail from 74.00 to 83.80. The paper interprets these gains as evidence that synthetic stateful MCP tasks teach transferable orchestration: planning, valid tool calls, result reading, and error recovery, rather than memorization of one fixed interface.
Coding Agent and Search Agent
The Coding Agent exposes one action: execute a shell command inside a persistent sandbox. The model must inspect files, edit code, run tests, and recover from failures over tens to hundreds of turns. Harbor provides a common directory format with an instruction, Dockerfile, and test script. Images are pre-baked into snapshot templates so rollout creation does not require a cold build.
Terminal-Bench 2.1 improves from 38.76 to 40.17 and SWE-bench Verified from 65.60 to 67.80. This was a compute-limited run, not a converged result. More importantly, the SWE-bench gain disappears by the shipped checkpoint, which returns to 65.6, while Terminal-Bench reaches 42.7. Since both tasks were not measured after every later stage, the paper cannot assign those final changes to Search Agent or RLHF individually.
The Search Agent operates on the open web with search, page extraction, and an isolated Python tool. Training allows up to 100 tool calls within a 128K trajectory; evaluation allows 200. Questions answerable without tools are removed, as are groups solved by all eight initial rollouts or by none, leaving about 2.2K questions with useful reward variation.
The judge scores answer quality, but positive reward also requires at least two successful tool calls. Malformed calls reduce the score, and exhausting the budget without a final answer incurs -0.1. A compact expression is:
$$R_i=s_i(1-0.5m_i)$$Here $s_i$ is the answer score and $m_i$ is the malformed-call fraction. Since the judge is noisy, Search Agent uses a mean-only group advantage rather than standard-deviation normalization:
$$\widehat{A}_i=r_i-\frac{1}{G}\sum_{j=1}^{G}r_j,\qquad G=16$$Normalizing by standard deviation would map a one-success group and a balanced group to similar advantage magnitudes, regardless of the actual reward difference. The paper reports a paired ablation in which the mean-only run wins at every matched validation point.
Search RL adds 3.0 on BrowseComp, 5.4 on Seal-0, and 3.4 on HLE-Verified. Training uses an internal search backend while evaluation uses a separate standardized stack. The transfer across backends is consistent with improved research behavior rather than exploitation of one search API.
RLHF: Place the Softest Reward Last
The final stage uses Skywork-Reward-V2-Qwen3-8B as an open reward model and trains on 75,815 filtered HH-RLHF prompts. The chosen and rejected responses in the source data are used only for offline analysis; the online policy is scored on its own rollouts. A linear length penalty reduces the reward of overlong positive-score answers without modifying negative rewards.
RLHF is last because a learned preference score offers the richest surface for reward hacking. Earlier placement would expose the policy to a brittle proxy for many later stages. Arena-Hard v2 Hard Prompt rises from 83.06 to 89.05, while Creative Writing rises from 38.56 to 52.97.
The same table also defines the boundary of the recipe. Rufus-Air remains behind GLM-4.5-Air on Creative Writing, 53.0 versus 60.3. A general preference model is not a substitute for a reward model tuned to prose style, and the paper does not claim otherwise.
Infrastructure Is Part of the Algorithm
Rufus-Slime runs SFT and general RL on Megatron-LM and SGLang. MoE sampling uses FP8 while parameter updates remain BF16. Rollout Routing Replay records routing decisions from generation and replays them during the training forward pass. Large rollout batches, truncation monitoring, and long-sequence bucketing are not incidental implementation details; they change which samples produce stable gradient estimates.
Rufus-Gym generalizes the agent loop across MCP servers, shell tasks, and search. Each environment consumes a task and returns the token trajectory, reward, and typed termination reason. The system passes token IDs, log probabilities, loss masks, and expert assignments directly from SGLang to the trainer. Re-rendering and re-tokenizing the full conversation at every turn is forbidden because it can alter chat templates, sampled tokens, and tool calls.
Coding trajectories need persistent files, dependencies, and processes. Each rollout therefore owns a Firecracker microVM, with pre-initialized state restored from a snapshot. This separates untrusted policy code through hardware virtualization instead of sharing a host kernel across thousands of concurrent workloads. E2B supplies the sandbox API, and Ray distributes CPU-heavy actor processes.
The engineering details extend to recovery. Commands are non-idempotent, so the system retries only when it can prove execution never started. If a command may have begun, the environment returns a recoverable error instead of dispatching it twice. This avoids duplicate writes and makes reward computation depend on a trace the model actually produced.
What the Final Model Shows
| Benchmark | Public GLM-4.5-Air | Rufus-Air | Change |
|---|---|---|---|
| IFBench | 33.6 | 76.9 | +43.3 |
| IFEval | 83.0 | 95.4 | +12.4 |
| LiveCodeBench v6 | 59.6 | 76.4 | +16.8 |
| MCP-Atlas | 35.9 | 45.3 | +9.4 |
| BrowseComp | 22.7 | 37.1 | +14.4 |
| Seal-0 | 33.3 | 51.4 | +18.1 |
| Terminal-Bench 2.1 | 24.7 | 42.7 | +18.0 |
| SWE-bench Verified | 50.6 | 65.6 | +15.0 |
The gains align with where the recipe spends stages and training signal. Instruction following, coding, search, and tool use receive most of the pipeline, and those areas move most. Mathematics and knowledge are handled primarily by one reasoning stage, so the final model is competitive rather than uniformly superior.
| Stage | Target metric | Before | After |
|---|---|---|---|
| Reasoning RL | GPQA | 68.18 | 73.50 |
| Coding RL | LiveCodeBench pass@1 | 68.6 | 75.9 |
| Instruction RL | Multi-challenge | 31.1 | 55.8 |
| General Agent | Tau2-Retail | 74.00 | 83.80 |
| Coding Agent | SWE-bench Verified | 65.60 | 67.80 |
| Search Agent | BrowseComp | 34.2 | 37.2 |
| RLHF | Arena-Hard v2 CW | 38.56 | 52.97 |
These numbers should be read as adjacent-checkpoint claims, not as interchangeable entries in one leaderboard. The stage-specific LiveCodeBench protocol differs from the final table. Tau2-Retail uses different user simulators in different tables. Search results can also vary slightly across evaluation versions. The paper documents these breaks instead of silently merging them.
Limitations
The authors state that the serial order is one sequence that worked, not one shown to be optimal. It came from prior experiments and available compute. They did not compare it with training domain specialists in parallel and merging or distilling them, so the reward-reliability principle remains a useful engineering heuristic rather than a fully isolated law.
Coding Agent stopped at the available compute rather than convergence. Its SWE-bench gain does not survive to the shipped checkpoint. No single benchmark suite was followed through every stage, so a difference between the Coding Agent checkpoint and the final model spans multiple later stages and cannot be assigned to one.
Search rewards remain noisy. Requiring two successful tool calls proves execution happened, not that the final answer depended on the returned evidence. The authors audit trajectories for incidental tool use, but the reward does not close that loophole completely. Search and page extraction also create real external costs that are separate from GPU training.
An independent reading should distinguish a documented recipe from a one-command reproduction. SFT alone uses 512 H200 GPUs for about 13 days. The coding sandbox service is sized for roughly ten thousand concurrent environments and costs about $10K per month. Search evaluation can generate hundreds of thousands of external calls. The data are public, but filtering, environment repair, and distributed infrastructure remain substantial engineering work.
There are also minor measurement discrepancies across stage tables. For example, the IF RL checkpoint is reported at GPQA 71.1 in one table and 73.36 in another. The paper explains that evaluation conditions differ. This supports reading Rufus-Air as a transparent systems report, where the broad trends and failed assumptions are more reliable than every isolated decimal.
Conclusion
Rufus-Air replaces the question “which RL algorithm should we use?” with a larger set of coupled decisions. SFT determines the starting capability, learnability filtering determines where gradients come from, reward reliability determines ordering, and token-faithful environments determine whether long-horizon agent training can scale. The reported endpoint is strong, but the transferable lesson is the structure that produced it.
For robotics foundation models and embodied agents, the implication is direct. A training team needs more than demonstrations and a scalar reward. It needs stateful environments that reflect real actions, failure labels that separate policy errors from infrastructure errors, and trajectories that preserve the exact tokens and observations seen by the model. Without those pieces, agentic RL can optimize noise or exploit a simulator rather than learn useful behavior.
The ceiling of post-training is set not only by how quickly a model learns, but by whether the recipe connects the right experience, a trustworthy reward, and a dependable environment into one coherent loop.
SOURCE LINKS
