Reinforcement Learning Enables Scaling Generalist Robot Policy Improvement

Paper: Reinforcement Learning Enables Scaling Generalist Robot Policy Improvement

Authors: Michal Korniak, Marco Bagatella, Ralf Römer, Michal Nauman, Yarden As, Pranav Vignesh, Manan Agarwal, and collaborators from ETH Zurich, MPI for Intelligent Systems, the Technical University of Munich, the University of Warsaw, Carnegie Mellon University, UC Berkeley, Amazon FAR, and Physical Intelligence.

Version: arXiv:2611.06905v1, November 4, 2026.

Resources: paper · project page · repository

Code status: The public repository contains a working FSFT path, a Best-of-N critic path, the multi-task collection loop, and adapters for MolmoSpaces and LIBERO. Its README says that the OGPO implementation is still on the way. The OGPO results therefore have to be read from the paper and appendix; they cannot yet be treated as a fully reproduced open-source result.

One-Sentence Summary

The paper turns online post-training of a vision-language-action policy from a single-task expert exercise into a multi-task generalization test: with a sufficiently diverse task distribution, critic-guided improvement lifts the base π0.5 policy by up to 58 percentage points on unseen scenes and 39 points on unseen objects, while direct successful-trajectory SFT often forgets.

Why This Problem Matters

Generalist robot policies already absorb broad visual, linguistic, and behavioral priors. Models such as π0.5, Octo, and OpenVLA can attempt many household tasks without task-specific training. Yet a policy that can attempt many tasks is not necessarily reliable enough for deployment. Robots still need to improve from their own experience after they encounter a particular kitchen, object set, or failure mode.

Most online robot learning is organized around one target task. A deployment collects trajectories for that task, updates the policy on its successes, and measures the gain on the same task. This recipe is easy to interpret, but it wastes the principal advantage of a generalist model: shared representations and skills across tasks. If every object requires a separate fine-tuning run, the total sampling and compute cost grows linearly with the deployment catalog, and improvements on one task may erase behavior learned on another.

The authors ask a stronger question. Can one online post-training run improve many tasks simultaneously and produce gains on tasks that were not part of the online distribution? They distinguish two forms of extrapolation. The first keeps the object fixed but changes the scene, testing visual robustness and spatial generalization. The second changes the object itself, often requiring a different grasp or motion strategy. A single average success rate would hide the difference between these cases, so the paper constructs separate controlled evaluations.

The title highlights reinforcement learning, but the study is not a simple RL-versus-SFT bake-off. The authors separate three mechanisms. Filtered supervised fine-tuning updates the policy only on successful trajectories. Best-of-N freezes the policy and learns a critic that ranks sampled action chunks at test time. Off-policy generative policy optimization, or OGPO, trains a critic and then extracts a better policy from its value estimates. All three improve in-distribution performance; their behavior outside the training distribution is where the scientific result appears.

This is primarily an empirical paper. It does not claim that a radically new optimizer solves generalist improvement. Its contribution is a controlled protocol that holds the base policy, rollout loop, and broad compute budget fixed while varying task diversity and the source of policy improvement. That choice makes the conclusions narrower, but also more defensible than a headline benchmark score with many uncontrolled differences.

Problem Formulation

A task is denoted by $c$, a state by $s$, an action chunk by $a$, and the policy by $\pi$. The state-action value function under task $c$ is the expected discounted return:

$$Q^{\pi}(s,a,c)=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_t,a_t,c)\mid s_0=s,a_0=a\right].$$

This quantity answers a practical question: if the robot executes this action chunk now and follows the policy afterward, how much long-horizon reward should it expect? The paper uses a time-to-success reward of $-1$ per step and $0$ on success. A higher Q value therefore means that the action leaves the robot closer to task completion, while a lower Q value means that it is likely to consume more time before success.

The test tasks are drawn from $\rho_{\mathrm{test}}$, while online interaction is available only on tasks from $\rho_{\mathrm{train}}$. The objective is:

$$J(\pi;\rho_{\mathrm{test}})=\mathbb{E}_{c\sim\rho_{\mathrm{test}},s_0\sim\mu_0(c)}\left[V^{\pi}(s_0,c)\right].$$

The important case is $\rho_{\mathrm{test}}\neq\rho_{\mathrm{train}}$. If the two distributions match, this is ordinary multi-task adaptation. If they differ, the method must transfer something beyond memorized task labels, such as an action-ranking rule, a grasping strategy, or a representation of task progress.

Online adaptation proceeds in rounds. In each round, the robot collects $K=20$ episodes for every training task, pools them in a replay buffer, and performs gradient updates. The standard run uses $M=10$ rounds. The largest 144-task BoN and OGPO runs use 15 rounds because their curves have not yet plateaued. This repeated collect-update protocol is called generalist improvement: the policy is not simply fit to a fixed offline dataset, and it is not specialized to one deployment task.

Three Mechanisms for Improving a VLA

1. Filtered SFT: Learn Only from Success

The first mechanism is weighted supervised learning. The policy minimizes:

$$\mathcal{L}_{\mathrm{DPA}}(\theta)=-\mathbb{E}_{(s,a,c)\sim\mathcal{D}}\left[w(s,a,c)\log\pi_{\theta}(a\mid s,c)\right].$$

For FSFT, the weight is a success indicator: state-action pairs from successful episodes receive weight one, and the rest receive zero. Modern flow-matching policies do not expose a simple Gaussian likelihood, so the implementation uses the corresponding flow-matching objective. The mechanism nevertheless remains close to behavior cloning.

FSFT is a useful baseline because it isolates direct policy adaptation. It changes the VLA using real successes but has no learned value function to compare two actions from the same state. In a single task, successful trajectories are usually concentrated enough that this filtering works well. In a diverse task set, different objects and scenes require conflicting actions. Increasing the probability of every successful action does not tell the model which action is better for the current object, nor how far a failed action was from success.

2. Critic Learning: Turn Sparse Success into an Action Ranking

The critic estimates the long-horizon value of a state-action-task tuple. The paper learns a Q function and a state-value function with coupled temporal-difference losses:

$$\mathcal{L}_{Q}(\psi)=\mathbb{E}_{(s,a,r,s',c)\sim\mathcal{D}}\left[\left(Q_{\psi}(s,a,c)-\left(r+\gamma V_{\eta}(s',c)\right)\right)^2\right],$$ $$\mathcal{L}_{V}(\eta)=\mathbb{E}_{(s,a,c)\sim\mathcal{D}}\left[\left(Q_{\psi}(s,a,c)-V_{\eta}(s,c)\right)^2\right].$$

The Q network learns how valuable an action chunk is from a particular state. The V network learns the average value of the current state under the data-collecting policy. During TD training, the next-state value supplies a bootstrapped target; during value learning, V regresses Q at actions stored in the buffer. Because the reward measures time to success, the critic can compare two imperfect actions even when neither episode eventually succeeds.

The critic does not consume raw images. It receives a mean-pooled prefix representation from the frozen π0.5 backbone, the robot's proprioceptive state, and the flattened action chunk. In BoN, the policy is frozen, so the prefix representation can be computed once at collection time and cached. Only the smaller residual value network is trained afterward. This is a major reason that critic-based improvement can be cheaper than repeatedly fine-tuning the full VLA.

The architecture is a BroNet residual MLP. Each block applies a linear layer, layer normalization, an activation, a second linear layer, and another normalization before adding the input. The public implementation captures the essential structure in src/rl/networks/bronet_critic.py:

class _BroNetBlock(nnx.Module):
    def __call__(self, x, activation):
        res = self.linear1(x)
        res = self.norm1(res)
        res = activation(res)
        res = self.linear2(res)
        res = self.norm2(res)
        return res + x

The residual path is not cosmetic. The ablation reports more than a 10 percentage-point gain over a standard MLP critic. The default critic width is 1024, its depth is 2, and its learning rate is $10^{-4}$. An ensemble is used for robustness, with the minimum value selected for conservative targets and action ranking.

3. Best-of-N: Improve Behavior without Changing the Policy

BoN keeps all π0.5 parameters frozen. At each decision point, it draws $N=32$ action chunks from the policy and executes the one with the highest target-critic value:

$$i^{\star}=\arg\max_{i}Q_{\mathrm{targ}}(s_t,a_t^{i},c),\qquad a_t^{i}\sim\pi_{\theta}(\cdot\mid s_t,c).$$

The policy can still generate poor actions. The bet is that among 32 samples, the critic can identify a better one. Online experience changes behavior through the learned ranking function rather than through gradients on the VLA. If the critic is calibrated, this is a conservative way to exploit new data without destroying pretrained behavior.

The source code exposes the complete path. Environments with the same prompt are grouped so that candidates can be generated in one batch. The learner samples all chunks, constructs the critic observation from normalized state and cached prefix tokens, scores every candidate, and selects the maximum.

candidates = self._sample_candidates(
    processed_obs, rng, n_samples
)
raw_state = _raw_state(processed_obs)
critic_obs, prefix = self._critic_observation(
    processed_obs, raw_state, policy_model,
    n_samples, droid_layout=droid_layout,
)
scores = self._score_candidates(
    q_model, critic_obs, candidates,
    raw_state, n_samples,
    droid_layout=droid_layout,
)
return _select_best(candidates, scores), prefix

This is the _best_of_n method in src/rl/best_of_n/best_of_n_learner.py. It also exposes the inference cost. Sampling 32 candidates and scoring each one is not free. The paper measures 3.49x the wall-clock latency of a single action query. Reducing N to 8 costs 1.59x and performs nearly as well, making it the more plausible deployment choice on a physical robot.

4. OGPO: Distill the Critic Back into the Policy

BoN changes behavior only while the critic is attached. The third approach asks the policy itself to produce higher-value actions. The abstract extraction objective is:

$$\mathcal{L}_{\mathrm{CPE}}(\theta)=\mathbb{E}_{(s,c)\sim\mathcal{D}}\mathbb{E}_{a\sim\pi_{\theta}(\cdot\mid s,c)}\left[Q_{\mathrm{targ}}(s,a)\right].$$

For a simple Gaussian policy, one might differentiate through the sampled action. A flow-matching VLA generates an action chunk through several denoising steps, which makes direct backpropagation through Q awkward and unstable. OGPO treats the denoising process as a small generative MDP. For every replay state, it samples a group of complete denoising trajectories and computes group-relative advantages:

$$\widehat{A_i}=Q_{\mathrm{targ}}(s,a^{i,(0)})-\frac{1}{G}\sum_{j=1}^{G}Q_{\mathrm{targ}}(s,a^{j,(0)}).$$

The question is no longer "is this action absolutely good?" but "is this action better than the other candidates sampled for the same state?" The method then forms an importance ratio for each trajectory using the current policy and a slowly updated reference policy. A PPO-style clipped objective increases the probability of positive-advantage trajectories and decreases the probability of negative-advantage trajectories:

$$\mathcal{L}_{\mathrm{OGPO}}(\theta)=-\mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G}C_{\epsilon}\left(\omega_i(\theta),\widehat{A_i}\right)\right].$$

The ratio $\omega_i(\theta)$ compares the likelihood of the denoising path under the current policy with its likelihood under a slower policy. Clipping prevents a small number of noisy value estimates from causing a large policy jump. The paper also keeps a behavior-cloning loss on successful transitions and uses conservative advantages when the critic is underfit.

On the critic side, OGPO reduces target variance by averaging the next-state Q over $N_{vr}=16$ actions sampled from the current policy:

$$\mathcal{L}_{\mathrm{critic}}=\mathbb{E}\left[\left(Q(s_t,a_t)-\left(r_t+\gamma\frac{1}{N_{vr}}\sum_{i=1}^{N_{vr}}Q(s_{t+1},a_{t+1}^{(i)})\right)\right)^2\right].$$

Mechanistically, OGPO combines direct policy adaptation with the critic comparison used by BoN. The critic is a temporary teacher rather than a permanent test-time attachment. If extraction succeeds, the policy can produce better actions with a single inference path, avoiding the 32-sample cost of BoN. The paper finds that this combination generally has the highest success rate while retaining the scaling behavior of critic-guided methods.

5. The Paper-Specific Pipeline

flowchart TB
    A[π0.5 online rollouts
K=20 per task] --> B[Shared replay buffer]
    B --> C[FSFT
success-only SFT]
    B --> D[BroNet critic
Q and V]
    D --> E[BoN
rank N=32 samples]
    D --> F[OGPO
group advantage and PPO]
    F --> G[Updated action expert]
    C --> H[Seen-task evaluation]
    E --> H
    G --> H
    H --> I[Unseen scenes]
    H --> J[Unseen objects]
    I --> K[Transfer depends on
task-diversity direction]
    J --> K

The diagram mirrors the actual experimental split. A shared online buffer feeds three improvement mechanisms, and evaluation separates scene extrapolation from object extrapolation. The result is not a generic claim that more tasks always help. The type of diversity determines which test axis improves.

Experimental Design

The main benchmark is the MolmoSpaces pick suite. It includes 18 objects and 55 to 56 scenes per object, creating a structured space of household manipulation tasks. Training sets contain 1, 4, 16, 64, or 144 tasks. Each larger split is a superset of the smaller one and adds both objects and scenes. The authors balance the splits so that the base policy starts near 30% success in every training set.

Each split is evaluated in three ways. In-distribution evaluation uses the training tasks. Unseen-scene evaluation holds out eight scenes for each training object. Unseen-object evaluation uses a fixed set of six objects that never appear in any training split, with eight scenes per object. A separate ablation varies scene count with one object and object count with one scene, allowing the authors to measure the direction of transfer.

LIBERO-90 provides an out-of-family check. FSFT and BoN are trained on nested sets of 1, 4, 16, and 64 tasks. A disjoint set of 25 tasks measures completely novel-task generalization. Initial success rates range from 23.4% to 27.0%, and the unseen tasks average 26.9%. LIBERO matters because its result does not support a broad positive claim: at the evaluated scale, neither method generalizes positively to completely new tasks.

Every experiment uses three seeds and reports 95% stratified bootstrap intervals. Each collection round gathers 20 episodes per task, and final evaluation uses 32 episodes per task. A single-task run takes roughly 2 to 5 wall-clock hours. The largest multi-task run takes about 200 hours on a four-GH200 node.

MolmoSpaces split	Tasks	Initial seen success	Initial unseen-scene success	Construction
1 task	1	30.2%	30.0%	One object and scene
4 tasks	4	30.2%	30.0%	2 objects x 2 scenes
16 tasks	16	29.2%	29.9%	4 objects x 4 scenes
64 tasks	64	29.5%	29.9%	8 objects x 8 scenes
144 tasks	144	29.0%	29.7%	12 objects x 12 scenes
Fixed unseen objects	6 objects x 8 scenes	Approximately 28.9%	Not applicable	No object occurs in training

The balanced initial success is essential to the interpretation. A curve that rises with task count is not merely selecting an easier split. Even the unseen-object set starts near the same success level, so gains after post-training reflect transfer from the training distribution rather than a different starting point.

Main Results

On the training tasks, all three methods improve. In the single-task setting, OGPO leads FSFT by 53 percentage points and BoN leads FSFT by 26 points. In multi-task settings, OGPO retains a 43 to 50 point advantage over FSFT, while BoN retains 15 to 21 points. The immediate conclusion is that critic-guided value estimation adds something that successful-trajectory SFT does not.

The unseen-scene curves show a transition from forgetting to transfer as task diversity grows. With a small number of tasks, direct policy adaptation can degrade behavior outside the training scene. At larger scales, BoN and OGPO improve over the base policy by up to 58 points on held-out scenes of the same objects. Scene changes primarily alter visual appearance and spatial arrangement, so the critic appears to learn a reusable notion of task progress rather than a fixed action sequence for one room.

The unseen-object result is stronger. On some objects never used for online training, the critic-guided policies beat the base policy by up to 39 points. This is not uniform object-level magic. Per-object results show that transfer follows geometry and grasp similarity. A model trained with boxes transfers better to bowls, and spatula training transfers better to knives. Objects with distant shapes and affordances remain difficult.

Atomizers and wine bottles provide a useful contrast. At the 144-task split, which adds bottle-like training objects, BoN improves atomizer success from 0.29 to 0.42 and OGPO improves it from 0.42 to 0.58. On the wine bottle, BoN rises from 0.24 to 0.44, but OGPO moves only from 0.09 to 0.12. When the target is far from the training distribution, freezing the policy and reranking candidates can be safer than extracting a new policy from an imperfect critic.

What Kind of Diversity Transfers?

The authors separate task diversity into two axes. The scene sweep fixes one object and varies the number of scenes from 1 to 4, 8, and 16. The object sweep fixes one scene per object and varies the number of objects from 1 to 4 and 8. Both sweeps are evaluated on the same unseen scenes and unseen objects.

Transfer follows the axis of diversification. BoN gains 36.5 points on unseen scenes from the scene sweep, and FSFT gains 31 points, but neither transfers well to new objects. Scaling objects eventually improves both axes. At a matched budget of eight tasks, the scene axis is 6.6 points better on unseen scenes, while the object axis is 9.4 points better on unseen objects.

The asymmetry is more important than the exact point estimates. If deployment variation is mostly a new room layout, collecting more scenes may be enough. If the goal is to handle new objects and grasps, the training set must contain different manipulation solutions. The paper therefore treats object diversity as the harder axis for generalist improvement, because it changes the action distribution rather than only the observation distribution.

Finding	Reported result	Interpretation
Single-task training improvement	OGPO +53 pp over FSFT; BoN +26 pp	Value guidance helps even without task diversity
Multi-task training improvement	OGPO +43 to +50 pp; BoN +15 to +21 pp over FSFT	The advantage persists as tasks grow
Unseen scenes	Up to +58 pp over base	Object held fixed; tests layout and visual transfer
Unseen objects	Up to +39 pp over base	New object requires transfer beyond the training tasks
Gradient efficiency	16-task run needs 2.9x fewer steps than 16 single-task runs	Shared learning amortizes post-training cost
Scene sweep	BoN +36.5 pp; FSFT +31 pp on unseen scenes	Scene diversity transfers mainly to scenes
Object sweep	Matched-budget +9.4 pp on unseen objects	Object diversity is the more general axis
BoN inference cost	3.49x for N=32; 1.59x for N=8	The performance gain has a real latency price

The efficiency result also changes how one should evaluate generalist improvement. A multi-task policy may be slightly worse than a collection of separate single-task experts on each training task, but it can reach a similar aggregate improvement with far fewer gradient steps. The relevant comparison is therefore not only final average success; it is also the cost of improving an entire deployment distribution.

Ablations and Resource Allocation

For direct policy adaptation, the authors compare full VLA fine-tuning with updating only the action expert. Full tuning is more than 12 points better in-distribution and on unseen scenes. It also degrades unseen-object performance by 10.6 points relative to the base policy, compared with only 2.2 points for expert-only tuning. The visual-language backbone provides task conditioning and adaptation capacity, but updating it increases the risk of overfitting the training distribution.

For value learning, BroNet beats a standard MLP by more than 10 points. TD targets with simple regression are the most consistent choice. Distributional critics perform poorly in multi-task settings, likely because return distributions differ across tasks. The result is a warning against carrying single-task architecture choices into multi-task VLA training without re-tuning.

For policy extraction, OGPO and AWR both outperform a straightforward FSFT plus BoN combination. The joint method has the right ingredients but not the same tight coupling between value optimization and policy likelihood. OGPO's clipped denoising-trajectory update gives the policy a direct training-time signal, while FSFT plus BoN leaves the critic outside the policy and changes behavior only at inference.

The data and compute study is equally relevant to practitioners. More episodes per round give diminishing returns. More gradient steps can hurt because the online RL dynamics are not perfectly stable. When data and compute are fixed, splitting the budget across more short iterations is better than collecting everything once and applying a long update phase. The standard protocol settles on 10 rounds with 20 episodes per task, while the large split uses 15 rounds because it has not saturated.

Critic pretraining is not a free accelerator. A critic pretrained on 64 MolmoSpaces tasks starts above a randomly initialized critic, but the advantage disappears within one iteration and then reverses on both unseen scenes and objects. The authors suggest a possible loss of plasticity but leave the mechanism unresolved. For engineering, this means that a pretrained value model should not be assumed to improve online adaptation merely because it has seen more tasks.

Design choice	Control	Observed outcome
VLA update scope	Full VLA vs action expert only	Full tuning +12 pp or more in distribution; -10.6 pp on unseen objects
Critic architecture	BroNet vs MLP	Residual BroNet improves by more than 10 pp
Value objective	TD regression, MC, classification, distributional	TD regression is most stable; distributional critics scale poorly
Policy extraction	OGPO, AWR, FSFT+BoN	OGPO and AWR generally beat the loose combination
Data scheduling	Single long update vs repeated short iterations	More frequent online iterations are better
Critic initialization	64-task pretrained vs scratch	Initial gain fades and reverses

LIBERO as a Boundary Case

MolmoSpaces is designed to make task diversity controllable, but that also means its training and test tasks remain inside one pick family. LIBERO supplies a harder test. Its unseen tasks differ in layout, objects, goals, and motion patterns, and are often not related by a single controlled axis. The paper trains on 1, 4, 16, or 64 tasks and evaluates on 25 disjoint tasks.

Both FSFT and BoN improve the training tasks. Neither produces positive generalization on the completely new tasks at this scale. BoN is still preferable because it barely forgets, while FSFT degrades substantially. The contrast with MolmoSpaces is the central boundary condition: value-based RL does not create arbitrary task generalization from unrelated data. It transfers along structure that exists in the training distribution.

This finding connects back to the wine-bottle result. When the target is close enough to the training set, extracting a critic-guided policy can be powerful. When the target is far away, changing the policy may damage useful pretrained behavior, whereas a frozen policy with a conservative critic can preserve competence. The paper does not provide a reliable test for choosing between these regimes before deployment; it only demonstrates that the distinction is real.

Limitations

The authors explicitly limit the study to manipulation. MolmoSpaces and LIBERO have clear task boundaries, short action chunks, and unambiguous success detectors. The evidence does not establish that the same critic learning will scale to locomotion, mobile manipulation, or long-horizon whole-body control. Those settings introduce changing contact, navigation, and credit assignment over much longer horizons.

The compute cost is also a real limitation. A single-task run takes 2 to 5 hours, while the largest multi-task run takes roughly 200 hours on four GH200 GPUs. The paper shows that shared multi-task training is more gradient-efficient than separate experts, but it does not show that the full recipe is affordable for ordinary robot fleets. Best-of-N adds another deployment cost: 32 candidates take 3.49 times as long as one action query.

A reproducibility limitation comes from the code release. FSFT and BoN are available, but OGPO is described as forthcoming. The appendix specifies group-relative advantages, trajectory importance ratios, clipping, conservative advantages, success-buffer BC, and critic variance reduction, yet external users cannot inspect the exact implementation. The most important positive results therefore depend on a method that is not yet end-to-end reproducible from the public repository.

My independent concern is critic calibration on unfamiliar objects. The ensemble and minimum aggregation reduce overestimation, but the paper does not report a systematic ranking-error metric on unseen objects. A confident but wrong critic can make BoN select an infeasible action, and OGPO can turn that ranking error into a persistent policy bias. The relatively poor OGPO result on wine bottles is consistent with this risk.

Finally, the task axes are manually constructed. MolmoSpaces makes it possible to vary scenes and objects independently, but real deployments do not provide a clean label telling engineers which diversity axis is missing. Automatically detecting whether a fleet needs more visual scenes, more object geometries, or a new manipulation skill remains an open data-selection problem.

Conclusion

The paper's core contribution is a clear decomposition of generalist improvement. FSFT changes the policy using successful actions. BoN freezes the policy and uses a critic to rank sampled actions. OGPO trains a critic and distills its preferences back into the generative policy. All three improve the training distribution, but critic-guided methods are the ones that turn increasing task diversity into positive transfer to unseen scenes and, in favorable cases, unseen objects.

The practical message is that online robot post-training should be designed around the deployment variation that matters. More scenes help with new scenes; new objects and grasp geometries are needed to transfer to new objects. A generalist policy does not improve merely because the collection set is large. It improves when the task distribution exposes the value function and policy to the structure they are expected to generalize over.

The next step is to connect this empirical picture to deployment decisions. A system needs a way to estimate whether its critic is reliable enough to update the policy, whether a frozen-policy BoN fallback is safer, and which missing task axis will most improve the next deployment distribution. The paper establishes that these questions are measurable; it does not yet answer them automatically.

Golden Line

Generalist improvement is not making one task more familiar. It is teaching value to transfer across tasks: the policy may remain frozen, but the critic must have seen enough diversity to judge what progress means in a new world.