AgenticRoboRL: Bringing Coding-Agent Post-Training to Agentic Robotics

Central claim

Robot-agent post-training can reuse an existing execution harness, but it must preserve the exact model-call context under which each response was sampled. The planner should learn which tool, code fragment, or recovery action to emit, not relearn a controller that is already available.

The scope is deliberately narrow. Perception, motion planning, the visual-language-action executor, and tool semantics stay fixed. Only the parameters of Qwen3.5-9B change. This makes the measured gains a question of planner behavior rather than a hidden benefit from retraining the low-level policy or adding memory during evaluation.

Formal setup

Let a robot task be $\xi_i=(x_i,s_i^0)$, where $x_i$ is the instruction and $s_i^0$ is the initial environment state. The harness is denoted by $\mathcal H$. At planner turn $t$, the harness constructs the context from the task, observations so far, and earlier responses:

$$c_{i,t}=\Phi_{\mathcal H}(x_i,o_{i,\leq t},y_{i,\lt t})$$

The planner samples a harness-native response from that context:

$$y_{i,t}\sim\pi_\theta(\cdot\mid c_{i,t})$$

The harness executes the response and returns another observation until the episode ends. The optimization objective is the expected return of the joint planner-harness process over a task distribution:

$$\max_\theta J(\theta)=\mathbb E_{\xi\sim\mathcal D,\tau\sim P_{\theta,\mathcal H}(\cdot\mid\xi)}[R(\tau)]$$

This work deliberately uses a binary terminal reward:

$$R(\tau)=\mathbf 1[\text{task success}]$$

The planner is not told which intermediate action deserves credit. Instead, multiple attempts from the same task and initial state provide a relative success signal. This keeps the outer algorithm portable, but it makes learning sensitive to whether a group contains at least one successful and one failed rollout.

Training infrastructure: record the actual sampled context

A recording proxy sits between the planner and the robot harness. For each request it stores the context, optional image references, tool specifications, generated token IDs, and rollout log probabilities. Before optimization, the trainer renders the prompt again and compares its token sequence with the recorded one. A sample whose prompt does not match receives zero optimization weight.

This gate matters because a robot harness may transform history between calls. It can prune old images, drop earlier reasoning, shorten tool output, or change serialization to control context cost. If the trainer reconstructed the transcript only at the end of an episode, it could optimize a response under a context that was never actually used. The policy ratio would then no longer describe the sampled policy.

Each harness also supplies an adapter that reports five values per episode: task, initial-state identifier, attempt index, reward, and a validity flag. The first two define the group used for comparison, while the attempt index distinguishes rollouts. A zero reward from an honest task failure remains in the group. Infrastructure failure, such as a simulator crash or incomplete record, is excluded from optimization.

Stage one: supervised fine-tuning on successful trajectories

In RPent, Qwen3.8-27B acts as a teacher with both task-specific and global memory. The 9B student is not given those memory modules. The authors therefore reconstruct each turn from the student's point of view by removing memory-access calls and their results. They then detect whether the teacher response depends on the removed information.

A response is flagged when it mentions memory-related terms, repeats eight consecutive words from the removed memory text, or contains a number that appeared only in a removed result. The teacher's reasoning and explanatory text are rewritten against the student context. A judge checks the rewrite, while tool-call arguments remain unchanged. Only responses judged acceptable become target answers. Other turns may be retained only as history or dropped.

CaP-X uses the same harness configuration for teacher and student, so this memory-removal step is unnecessary. The pipeline collects model calls from successful episodes that ended normally, re-renders each prompt with the student chat template, and uses the teacher's recorded output tokens as the target.

Supervised loss is normalized over all target tokens in the optimizer batch rather than over independently averaged responses:

$$\mathcal L_{\mathrm{SFT}}=-\frac{\sum_{s\in\mathcal B_{\mathrm{SFT}}}\sum_{j=1}^{L_s}\log\pi_\theta(y_{s,j}\mid c_s,y_{s,\lt j})}{\sum_{s\in\mathcal B_{\mathrm{SFT}}}L_s}$$

Only response tokens receive loss, including reasoning, text, serialized tool calls or code, and the end token. Context tokens are masked. The global token denominator prevents long responses from receiving the same total weight as short ones simply because each response was averaged separately.

Stage two: multi-turn reinforcement learning

RL starts from the supervised checkpoint. One task and one initial state define a group, and each group is sampled six times. A group is used only if at least two attempts are valid. Let $R_i$ be the return of attempt $i$. The advantage uses the mean reward of all other attempts in the group:

$$A_i=R_i-\frac{1}{G_g-1}\sum_{k\in\mathcal V_g\setminus\{i\}}R_k$$

There is no learned critic and no reward-standard-deviation normalization. Every selected response in an episode inherits that episode's advantage. The model is therefore rewarded for the success of a multi-turn sequence, not asked to imitate a teacher answer turn by turn.

To control cost, at most 16 eligible responses are selected from each episode. Eligibility uses record validity, token length, and a post-success filter. If episode $i$ contributes $K_i$ selected responses, each response receives episode weight $u_s=1/K_{i(s)}$. The policy loss is:

$$\mathcal L_{\mathrm{RL}}=\frac{\sum_{s\in\mathcal B_{\mathrm{RL}}}\frac{u_s}{L_s}\sum_{j=1}^{L_s}w_{sj}\ell_{\mathrm{DC}}(\rho_{sj},A_{i(s)})}{\sum_{s\in\mathcal B_{\mathrm{RL}}}u_s}$$

The inner division by response length $L_s$ balances tokens, while the outer episode weight prevents long trajectories from contributing more than short ones. This is a two-level weighting scheme: episodes first, tokens second.

Stable updates: truncated importance sampling and dual clipping

Sampling and training can run through different numerical backends, so the implementation applies a truncated ratio between sampling probability $q_{sj}$ and pre-update probability $b_{sj}$:

$$w_{sj}=\operatorname{sg}\left[\min\left(\frac{q_{sj}}{b_{sj}},C\right)\right]$$

Here $\operatorname{sg}$ stops the gradient and the main configuration sets $C=2$. The policy ratio is formed against the recorded sampling probability:

$$\rho_{sj}=\frac{\pi_\theta(y_{s,j}\mid c_s,y_{s,\lt j})}{q_{sj}}$$

The clipped objective is further bounded on negative-advantage tokens:

$$\ell_{\mathrm{DC}}(\rho,A)=\begin{cases}\min\{\ell_{\mathrm{PPO}},-dA\}, & A \lt 0,\\ \ell_{\mathrm{PPO}}, & A\geq 0.\end{cases}$$

The main run uses $d=3$, with lower and upper clipping values of $0.2$ and $0.28$. Negative examples are bounded more aggressively, while successful responses can move probability upward within a slightly wider range. The released `rl_loss.py` implements these equations, and `check_gate_a.py` enforces token-identical prompt reconstruction before training.

Method data flow

flowchart LR
  T["Successful Qwen3.8-27B trajectories"] --> C["RPent memory-reference rewriting
CaP-X direct reuse"]
  C --> S["Student context and target response"]
  S --> F["SFT of Qwen3.5-9B planner"]
  F --> P["Multi-turn planner rollouts"]
  P --> H["RPent VLA calls
or CaP-X control code"]
  H --> E["Robot environment"]
  P --> R["Recording proxy
context tokens logprobs"]
  E --> O["Success reward
task failure or invalid state"]
  R --> M["Match calls to episode"]
  O --> M
  M --> G["Task and initial-state groups
leave-one-out advantage"]
  G --> U["Truncated importance sampling
dual-clipped policy update"]
  U --> P
  I["Infrastructure-invalid episode"] -. "excluded" .-> O
      

The feedback loop stops at the model boundary. Neither the controller nor the visual-language-action executor is trained. This is what permits reuse of existing harnesses, but it also means the planner cannot repair a low-level skill that is systematically incapable of executing a requested behavior.

Experimental setup

The main experiment uses RPent on LIBERO-PRO. The benchmark adds position swaps and task-instruction perturbations to four LIBERO suites, yielding eight evaluation suites with ten tasks and ten initial states each, or 800 episodes. Teacher demonstrations and RL rollouts use states 11 through 44. States 45 through 49 are reserved for validation and selection, while evaluation uses states 1 through 10. State 0 is used only to construct task memory under the comparison setting.

The student is Qwen3.5-9B and the teacher is Qwen3.8-27B. The student uses no external memory during RL rollouts or evaluation. The selected checkpoints are also evaluated on all 10,030 LIBERO-Plus configurations without additional benchmark-specific training.

The second harness, CaP-X, requires the planner to generate executable control code. The authors use the Object, Goal, and Spatial LIBERO-PRO suites under position and task perturbations, for 60 tasks, and keep the privileged-state setting used by the CaP-X RL experiment. Teacher, base planner, SFT planner, and RL planner all use the same interface, so the comparison isolates the planner rather than the control representation.

The first six RPent RL rounds contain 2,304 episodes, 26,052 selected responses, and 206 optimizer steps. An RPent episode is capped at 100 planner turns or 7,200 seconds. A CaP-X response is capped at 32,768 tokens. Sampling uses temperature 1.0; RPent evaluation additionally uses top-p 0.95, top-k 20, and a presence penalty of 1.5.

Main results: SFT supplies the foundation and RL improves it

On LIBERO-PRO, the original 9B planner succeeds on 18.1% of episodes. SFT raises this to 65.1%, and RL reaches 71.0%. Most of the 52.9-point total gain therefore comes from successful teacher supervision, while RL contributes another 5.9 points. The result is closer to learning harness usage in distribution first and then correcting the policy with outcome differences than to discovering every successful behavior from a weak random starting point.

The trained 9B planner exceeds the no-memory 27B planner at 57.9% and comes within 3.0 points of the memory-enabled 27B teacher at 74.0%. It is also close to Codex GPT-5.5 with memory at 72.1%. The authors correctly describe the latter comparison as system-level: a memory-enabled closed-source agent and a student without memory are not being used to isolate model scale.

Across the 800 paired episodes, RL solves 121 episodes that SFT fails and fails 74 episodes that SFT solves. The exact sign test gives $p=0.001$. The asymmetry is statistically informative, but the 74 regressions show that RL changes policy coverage rather than making every episode monotonically better.

LIBERO-Plus contains 10,030 configurations absent from training. Overall success rises from 19.2% to 66.7% after SFT and to 72.2% after RL. RL improves every perturbation category relative to SFT, from 2.2 points on object layout to 7.0 points on lighting. On identical evaluation instances, it succeeds where SFT fails 1,547 times and fails where SFT succeeds 999 times, for a net gain of 548 successful instances.

Generality from VLA skills to control code

CaP-X is much harder in absolute terms. The teacher succeeds often enough to provide data on only 26 of 60 tasks. Even so, SFT raises the 9B planner from 2.7% to 18.7%, and RL reaches 28.5%, above the 27B teacher's 20.3%.

The largest improvements are in the two Object suites, where RL raises success from 39.0% to 49.0% up to 67.0% to 72.0%. Goal and Spatial remain weak because the teacher rarely succeeds there. The RL training pool nevertheless includes nine tasks with no teacher data among its 35 tasks, so part of the improvement comes from task-success rollouts rather than additional imitation targets.

This experiment provides the stronger evidence for a shared post-training interface. The two harnesses expose different response formats, episode lengths, and filters, but the core advantage calculation and policy objective remain the same. At the same time, 25 of the 60 evaluation tasks are never used in training, and Goal and Spatial have sparse success, so the aggregate result does not establish broad cross-task generalization.

Ablation: rewriting teacher responses versus masking them

RPent supervision must handle teacher responses that refer to memory unavailable to the student. The mask-only recipe removes those responses. The rewrite-and-filter recipe reconstructs the student context, rewrites the reasoning, checks the rewrite, and keeps tool arguments unchanged. Mask-only yields 27,110 target turns, while rewrite-and-filter yields 40,047.

After one epoch, mask-only improves success from 18.1% to 55.2%, while rewrite-and-filter reaches 64.6%. Because the data sizes differ, the authors also compare similar update counts. Rewrite-and-filter reaches 60.2% at step 438, compared with 55.2% for mask-only at step 424. The advantage therefore does not appear only at the end of a longer training run.

Figure 3: A reader-facing plot of reported SFT checkpoints: no SFT at 18.1%, mask-only at 55.2%, rewrite-and-filter at 60.2% and 64.6%, and the selected step-563 checkpoint at 65.1%.

The authors explicitly note that this comparison does not isolate the effect of rewriting. The two recipes change both response content and supervision coverage. The 9.4-point end-of-epoch gap cannot be attributed entirely to rewrite quality. A stronger causal claim would require controls with matched target-turn counts, matched optimization steps, and matched task coverage.

Reproduction details visible in the released code

The public repository contains the recording proxy, round builder, policy loss, prompt gate, and RPent adapter inspected for this review. The proxy converts images into content-addressed hash references so large files are not duplicated, while preserving prompt tokens, output tokens, log probabilities, and tool specifications. The round builder checks validity, completeness, and policy version before constructing groups.

The adapter separates task failures from infrastructure failures and filters interactions after a task has already succeeded. This post-success filter supports the paper's choice of model calls as the training unit. Keeping a long sequence of unrelated calls after success would assign the successful episode's advantage to behavior that did not help achieve it.

The most important reproduction condition is not a network hyperparameter but context identity. The proxy, chat template, image references, tool serializer, and adapter must all preserve the prompt tokens shown to the sampled policy. The dedicated prompt gate in the repository makes this an explicit correctness requirement.

Why the result matters

The paper makes three contributions. It transfers coding-agent post-training to the planner layer of robot agents; it defines a model-call interface that supports both skill invocation and control-code generation; and it reports a reasonably complete comparison across two benchmarks, two harnesses, SFT, RL, and several training budgets.

The system implication is that a team can improve an existing robot-agent stack without retraining a costly low-level policy. By moving learning to tool and code selection, the planner can turn execution failures into recovery behavior while preserving the harness's safety and control mechanisms.

The research implication is equally direct. A 9B open planner after post-training reaches 71.0% without memory, close to a much stronger closed model with memory. New work that compares only base models or adds another memory layer now needs to explain why it does not also use successful robot interaction to update the planner.

Limitations

The authors explicitly state that the SFT ablation does not isolate rewriting alone. Rewrite-and-filter changes both response content and the number of supervised turns, while mask-only keeps a smaller set of unmodified responses. The observed advantage is useful, but a causal claim about rewrite quality would require better matched controls.

The reward is extremely sparse. Only final task success is optimized. There is no credit for approaching a goal, reducing collision, making a valid partial tool call, or completing a sub-task. If all six attempts in a group fail, the relative advantage is zero and no policy direction is learned from near misses.

The common pipeline still needs a harness-specific adapter. A new robot stack must route planner calls through the proxy, define stable task and initial-state identities, expose attempt numbers, and distinguish infrastructure failure from genuine task failure. This is less work than redesigning the learning algorithm, but it is not zero integration cost.

Evaluation is entirely in simulation. The experiments cover LIBERO-PRO, LIBERO-Plus, and CaP-X, but do not report a physical-robot success rate, intervention frequency, recovery statistics, or safety violations. The evidence supports improved benchmark planning, not long-term reliability in real homes, factories, or human-populated spaces.

Teacher coverage remains a bottleneck. CaP-X performance is much lower on Goal and Spatial, where the 27B teacher rarely succeeds and supplies little SFT data. RL can improve tasks without teacher labels, but it still requires at least one successful rollout in a comparison group. Extremely difficult tasks can therefore remain outside the reachable training distribution.

Finally, the public release is not complete relative to the paper. Code, the project page, and a trained checkpoint are available, but the teacher trajectories promised by the authors were not found at the time of this review. Their absence limits independent reproduction of the rewrite filters, sample counts, and exact supervision distribution.

Conclusion

AgenticRoboRL shows that improving a robot agent does not require starting from the low-level policy. By freezing perception, motion, and VLA execution, recording planner calls under their real contexts, initializing from successful teacher trajectories, and optimizing the same planner with task outcomes, the method produces substantial gains in both skill-calling and control-code harnesses.

The durable idea is the system boundary: keep execution components fixed while sharing the learning interface at model calls. The main open problems are sparse reward, limited teacher coverage, and validation on physical robots. Combining richer process signals with scalable success discovery and long-running real-world evaluation is the natural path from benchmark improvement to dependable deployment.

Original paper and code

The full paper is available at arXiv 2610.28589. The implementation is at L-CodingSpace/AgenticRoboRL, and the released planner checkpoint is hosted in the Hugging Face AgenticRoboRL-Qwen3.5-9B repository.