A curriculum for post-training agents

A curriculum for post-training agents

Source: A curriculum for post-training agents, Hugging Face, October 20, 2026.

A pretrained language model can continue a piece of text, but an agent has to do something different: inspect a task, choose a tool, read the result, revise its plan and continue until the task is finished. One answer is no longer the unit of behavior. A single episode may contain dozens of messages, several tool calls and observations supplied by an external process. Post-training an agent therefore means training over that complete interaction history, not only over a polished final response.

This curriculum divides agentic post-training into five stages. Training a coding agent is the running example because its actions can be executed and checked, but the same progression applies to data analysis, browser automation, operations work and simulation-based robotics. The stages move from the strongest and most expensive supervision to the weakest and most scalable signal:

1. Supervised fine-tuning (SFT) learns from complete traces produced by a stronger model.
2. Distillation transfers a teacher's behavior, either through generated traces or through probabilities on the student's own tokens.
3. GRPO learns from scores on multiple responses sampled for the same prompt.
4. RL environments turn that score into feedback from a stateful, multi-turn task.
5. Harnesses and asynchronous infrastructure place the real agent loop in a sandbox and keep generation and training from waiting on each other.

The central lesson is that the training interface should match the deployment interface. If the deployed agent uses a particular tool schema, context policy and stop condition, the training rollout should not silently replace them with a clean imitation. Otherwise the optimizer may learn a policy for a simulator that no user ever runs.

Steps at a glance

Stage	What provides the signal	TRL trainer	What you prepare	Main limitation
SFT	Correct agent traces	SFTTrainer	Prompt, full trajectory and an assistant-token loss mask	The model does not practice recovering from its own mistakes.
Distillation	Teacher outputs or teacher probabilities	DistillationTrainer	Prompts plus a teacher model	Student quality is bounded by the teacher and teacher inference is costly.
RL with GRPO	A score on sampled answers	GRPOTrainer	Prompts plus one or more reward functions	The reward is sparse and must be hard to game.
RL environments	Outcomes observed after several actions	GRPOTrainer with OpenEnv	Tasks, tools, state transitions and a final checker	Rollout generation becomes slow and operationally complex.
Harness at scale	Complete episodes run through the production-like loop	AsyncGRPOTrainer	Tasks, environment, harness and separate inference capacity	Staleness, sandbox isolation, failures and throughput must be managed.

These stages are a curriculum, not a mandatory sequence in which every project must pay for all five. A team with no usable policy may need SFT first. A strong teacher and a limited data budget can make distillation the best next step. If a reliable verifier exists, GRPO can optimize capabilities that demonstrations rarely show. A stateful environment becomes necessary when the outcome depends on actions rather than on one answer.

0. Before you start: separate the agent from its runtime

Before comparing trainers, it helps to make four terms explicit. The model is the learned policy. The agent loop decides when to call the model, parse its output and send the next observation. The harness is the executable scaffold around that loop: system prompt, tools, context management, parsers, retry rules and stop logic. The environment owns task state and decides what happens when an action is executed.

The boundaries are not universal, but they must be consistent inside one experiment. A useful checklist is:

• Which role produces the token loss: assistant only, whole conversation, or a tagged subset?
• Which observations are visible to the policy at each turn?
• Can tool calls fail, time out, return malformed output or require recovery?
• When is the episode considered finished, and who produces the final reward?
• Which parts are frozen across SFT, RL and evaluation?

Changing any of these between training and evaluation creates a distribution shift that is easy to miss. A model can look strong on a cleaned training trace and then fail when a real tool returns an error after five successful turns.

1. Supervised fine-tuning: learn from complete trajectories

SFT treats a recorded agent run as the target sequence. Each trace contains the task, the messages, the tool calls, the tool results and the model's later decisions. The model is trained to assign high probability to the actions that a stronger model or a human demonstrator took.

For a trajectory $i$ with assistant tokens $y_{i,t}$, a typical objective is the token-level cross-entropy below. The mask $m_{i,t}$ matters in agent training because prompts, tool schemas and environment observations should normally not be treated as actions that the model must predict.

$$L_{\mathrm{SFT}}=-\frac{\sum_{i,t}m_{i,t}\log\pi_\theta(y_{i,t}\mid x_i,y_{i,<t})}{\sum_{i,t}m_{i,t}}$$

The advantage is stability. Every target token is known, the loss is dense, and the same recipe scales to large offline datasets. It is also the cheapest way to teach format: when to call a tool, how to write arguments, how to interpret an observation and how to stop. For a small model that cannot reliably emit a tool call at all, SFT is often the necessary first stage.

The limitation is equally important: the student only sees successful demonstrations. It does not generate its own mistakes, so it never receives a learning signal for recovering from a failed shell command, a stale browser state or an invalid JSON response. This is the exposure-bias problem in its agentic form. At inference time the policy will make errors; at training time it was never asked to continue after one.

Good SFT data therefore includes more than final code. Keep the complete tool sequence, observation boundaries, retries and termination behavior. Remove hidden reasoning that the deployed policy will not have access to. If the trace was cleaned so heavily that it no longer resembles the runtime, the model is learning a different task.

2. Distillation: keep more information than hard labels

Distillation still uses a teacher, but it can expose the student to a richer signal. In the sequence-level form, the teacher generates trajectories and the student imitates them. In the token-level form, the teacher assigns probabilities to the student's own next tokens. The second form is on-policy: the student creates the context, and the teacher tells it how the next action should be distributed.

For teacher distribution $q_t$ and student distribution $p_\theta$ over vocabulary $V$, a conceptual distillation loss is:

$$L_{\mathrm{KD}}=-\sum_t\sum_{v=1}^{V}q_t(v)\log p_\theta(v\mid c_t),\qquad q_t=\operatorname{softmax}\left(\frac{z_t^{T}}{\tau}\right)$$

Here $c_t$ is the student-created context, $z_t^{T}$ are teacher logits and $\tau$ is a temperature. The teacher does not merely say "call run_tests". It can indicate that several actions are plausible and by how much. That dense supervision usually needs fewer samples than RL, where an entire trajectory may receive one score.

Distillation has two costs. First, the student cannot become more capable than the evidence supplied by the teacher, except through the student's own inductive biases and any additional on-policy exploration. Second, on-policy distillation must run the teacher during student rollout or training. Teacher inference can dominate the budget, especially for long agent episodes.

The practical choice is therefore not "distillation instead of RL". It is a choice about supervision density. Use demonstrations when a good trajectory is available. Use probability distillation when the teacher can grade the student's own choices and token-level guidance is worth the extra compute. Use reward-based RL when the right behavior is not easy to imitate but the final outcome can be checked.

3. Reinforcement learning: learn from outcomes with GRPO

RL removes the requirement for a teacher. The model samples answers for the same prompt, a reward function scores them, and the update makes better-scoring behavior more likely. The reward can be an exact checker, a test suite, a constrained model-based grader or a carefully validated combination of signals.

Hugging Face uses Group Relative Policy Optimization (GRPO) in this course. For each prompt, GRPO samples a group of $K$ responses and compares their rewards inside the group. No separate value model is needed to estimate a baseline. The group mean itself becomes the reference point.

$$A_i=\frac{r_i-\operatorname{mean}(r_1,\ldots,r_K)}{\operatorname{std}(r_1,\ldots,r_K)+\epsilon}$$

With policy ratio $\rho_t(\theta)$ and clipping threshold $\epsilon$, a standard GRPO-style objective has the familiar language:

$$\rho_t(\theta)=\frac{\pi_\theta(y_t\mid s_t)}{\pi_{\theta_{\mathrm{old}}}(y_t\mid s_t)}$$

$$L_{\mathrm{GRPO}}=-\mathbb{E}\left[\min\left(\rho_t(\theta)A_i,\operatorname{clip}\left(\rho_t(\theta),1-\epsilon,1+\epsilon\right)A_i\right)\right]+\beta D_{\mathrm{KL}}\left(\pi_\theta\Vert\pi_{\mathrm{ref}}\right)$$

The exact implementation may normalize the advantage differently, use a token-level or sequence-level ratio, mask tool observations or apply a variant such as DAPO, Dr. GRPO, GSPO or CISPO. The shared idea is still relative comparison: a group of attempts supplies enough information to tell which behavior should become more likely.

The signal is much weaker than token-level distillation. A complete answer or episode may collapse to one scalar, so the model needs many attempts. Sparse rewards also create a variance and exploration problem. If every sample gets the same score, the centered advantage becomes zero and the prompt contributes no gradient. If the reward is easy to exploit, optimization will find the exploit.

A practical reward stack therefore needs more than a headline objective. It often contains a correctness check, a format gate, safeguards against destructive actions, latency or step penalties, and a mechanism for detecting reward hacking. Each term changes the policy, so adding a reward is a design decision rather than a harmless metric.

4. RL environments: let the agent act more than once

A plain reward function usually scores a single response. An environment lets the agent act repeatedly, observe what changed and repair its own work. This is the difference between asking a model to write a patch and letting it inspect a repository, edit files, run tests, read failures and try again.

The interface descends from the Gym loop: reset a task, apply an action, receive a new observation and reward, and eventually mark the episode done.

observation, info = env.reset(seed=task_seed)
done = False
while not done:
    action = policy(observation)
    observation, reward, terminated, truncated, info = env.step(action)
    done = terminated or truncated

In an agent environment, the returned observation may be a shell result, a file diff, a browser screenshot, a simulator state or a test report. The environment owns the state transition. It can execute tools, enforce permissions and decide whether the final artifact passes.

A trajectory can be represented as:

$$\tau=(s_0,a_0,o_1,s_1,a_1,o_2,\ldots,s_T,a_T,o_{T+1})$$

With a learned reward function, the objective might be a discounted return:

$$R(\tau)=\sum_{t=0}^{T}\gamma^t r_t$$

Many agentic tasks instead use a terminal verifier: $R(\tau)=V(\tau)$, where $V$ runs tests, checks a database, validates a file or evaluates a simulator goal. The final check is often more trustworthy than a per-step model judgment, but it provides no gradient over the whole trajectory. Credit assignment remains difficult: which of thirty actions deserves credit for the final pass?

The original article points out that this loop is old, while the conditions that make it work are new. In 2016, a reset-step interface existed, but researchers lacked strong pretrained policies, a large supply of reachable tasks, reliable programmatic reward checks and cheap sandbox orchestration. Modern language models can act inside tools, modern clouds can start isolated machines per rollout, and many digital tasks expose an executable success condition. That combination is why environments are now practical rather than merely conceptually attractive.

Environment component	Responsibility	Agent example	Robotics example
State	Preserve what earlier actions changed	Repository, shell and browser session	Robot pose, object state and simulator clock
Action API	Define valid operations and arguments	Edit file, run command, click element	Joint command, grasp request, navigation goal
Observation	Return the result of an action	Test output, diff or page snapshot	Camera frame, proprioception or contact feedback
Termination	Decide when success or failure is final	Tests pass, timeout or budget exhausted	Goal reached, collision or episode limit
Verifier	Convert final state into reward	Run hidden tests	Measure goal distance and safety constraints

Long episodes make rollout generation the bottleneck. The trainer may spend seconds on a weight update and minutes waiting for an agent to finish a difficult task. Worse, a single infrastructure failure can invalidate a partially completed trajectory. Robust environments need timeouts, retries, resource limits, deterministic seeds where possible and a clear distinction between a task failure and a harness failure.

5. Harnesses and scale: train in the loop you will deploy

The last stage puts the full production-like harness inside a remote sandbox. The model does not receive a simplified transcript invented by the trainer. It runs through the same prompt assembly, tool parser, context manager, retry policy and completion checker that the real agent uses. Each rollout becomes a complete episode in an isolated machine.

This choice improves transfer but makes throughput harder. Generation and training have different hardware profiles and different latency patterns. If both run synchronously on one set of GPUs, the trainer sits idle while agents wait for tools, and the inference engine sits idle while weights are updated. Asynchronous RL separates the two pools and passes rollouts and weights between them.

flowchart LR
  D[Task dataset] --> R[Rollout workers]
  R --> H[Production-like harness]
  H --> E[Remote sandbox and tools]
  E --> H
  H --> B[Completed episodes buffer]
  B --> T[Policy trainer GPUs]
  T -->|Updated weights| V[vLLM inference GPUs]
  V --> R

The asynchronous loop introduces a new state variable: policy age. If a rollout was sampled under policy version $k_r$ while the trainer has already advanced to $k_p$, the sample is stale when it reaches the update.

$$\Delta_{\mathrm{policy}}=k_p-k_r,\qquad \Delta_{\mathrm{policy}}\leq S_{\max}$$

TRL's experimental AsyncGRPOTrainer exposes controls around this system. A separate vLLM server generates trajectories while the trainer updates on other GPUs. Parameters such as max_staleness, weight_sync_steps and max_inflight_tasks trade freshness against utilization. Fresh weights improve on-policy behavior; too frequent synchronization and too few in-flight tasks leave hardware idle.

A minimal trainer setup can be this short:

from trl.experimental.async_grpo import AsyncGRPOTrainer
from trl.rewards import accuracy_reward
from datasets import load_dataset

dataset = load_dataset("trl-lib/DeepMath-103K", split="train")
trainer = AsyncGRPOTrainer(
    model="Qwen/Qwen3-4B",
    reward_funcs=accuracy_reward,
    train_dataset=dataset,
)
trainer.train()

The inference server needs its own GPU allocation. A documentation example starts it in vLLM server-development mode with a supported log-probability mode and NCCL weight transfer:

CUDA_VISIBLE_DEVICES=0 VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3-4B \
  --max-model-len 4096 \
  --logprobs-mode processed_logprobs \
  --api-server-count 8 \
  --weight-transfer-config '{"backend":"nccl"}'

The short example demonstrates the interface, not a complete production recipe. A real run also needs sandbox lifecycle management, task queues, failure classification, reward auditing, checkpointing and monitoring of policy staleness. The expensive part is often not the optimizer. It is keeping thousands of heterogeneous episodes moving without corrupting state or silently scoring the wrong artifact.

How to choose the next stage

Current symptom	Better next stage	Reason
The model cannot produce the required tool format	SFT	Start with many clean demonstrations and mask non-assistant tokens.
The format works, but a teacher can explain better token choices	Distillation	Add dense supervision on student-generated contexts.
The final answer is easy to verify, but demonstrations are scarce	GRPO	Replace imitation with reward-driven exploration.
One-shot answers cannot solve the task	RL environment	Expose state, tools and a multi-turn success condition.
Training works on toy traces but not in deployment	Production harness	Train through the actual parser, prompt, tools and stop rules.
GPU utilization collapses while agents wait on tools	Async harness infrastructure	Decouple rollout generation from policy updates and bound staleness.

What this means for embodied agents

The same curriculum is useful for robot foundation models, even though the observation and action spaces differ. SFT can teach a policy to produce valid action chunks or tool calls from demonstrations. Distillation can transfer a larger multi-modal teacher into a smaller onboard policy. GRPO can optimize outcomes when a simulator, judge or task checker is available. An environment can carry forward robot state across perception, planning and manipulation. A harness can reproduce the exact context window, safety filter and tool API used on the robot.

The difficult parts also become clearer. Robot rollouts are slower and more expensive than shell commands. Simulators are approximate. A physically invalid action may damage hardware or create an unsafe state. The environment must therefore encode hard safety constraints separately from reward. Asynchronous training is valuable because simulator and policy inference can run in parallel, but stale weights are especially risky when the environment is unstable or the controller assumes fresh observations.

The organizing principle remains the same: move supervision as close as possible to the behavior that will actually be deployed. Demonstrations show the desired path. Distillation grades local decisions. Reward optimization discovers alternatives. Environments test the consequences of action. Harnesses make the training loop look like the real loop.

References and further reading

• A curriculum for post-training agents, the source article and curriculum overview.
• How frontier models train on outcomes in 2026, on verifiable rewards and modern RL recipes.
• Why RL environments work now, on the gap between the Gym era and modern executable tasks.
• Keep the Tokens Flowing, on asynchronous RL infrastructure across open-source systems.
• TRL SFT Trainer, Distillation Trainer, GRPO Trainer and Async GRPO Trainer.
• OpenEnv documentation for stateful agent environments and harnesses.

---

Source: A curriculum for post-training agents, Ben Burtenshaw, Sergio Paniego and Merve, Hugging Face, October 20, 2026. This RobotWorld edition expands the curriculum with implementation details, equations and an embodied-agent interpretation.