PAPER DEEP DIVE
Learning Multi-Phase Robot Loco-Manipulation via Phase-Conditioned Policy Optimization
Complex whole-body loco-manipulation requires executing diverse maneuvers, such as crouching, grasping, and locomotion, within a single long-horizon task. While performing simple pick-and-place behaviors is well-established, scaling to complex, long-horizon tasks remains difficult. Currently, this challenge is addressed by relying on human motion data and correspondingly humanoid platforms, which limits platform independence and introduces substantial data processing overhead. Instead, we extend proximal policy optimization (PPO) to train a single policy for multi-phase tasks across diverse robotic hardware, leveraging prior task structure to shape behavior. Our method adapts PPO using a phase-conditioned critic, phase-gated reward functions, and a multi-phase reset curriculum to guide exploration. To support varied hardware, we employ a decoupled hierarchical control architecture that separates lower-level balancing dynamics from high-level phase execution. We demonstrate multi-stage loco-manipulation skills on hardware on both the Unitree G1 and a Boston Dynamics Spot equipped with a manipulator arm, and validate our design choices through ablations. Our approach enables the acquisition of complex loco-manipulation behaviors purely through environment interaction while directly incorporating high-level task structure into policy learning.
Paper metadata: Simone Manni, Annika Wollschläger, Dario Bellicoso, Abel Gawel, Marco Hutter, Simon Le Cleac'h, and Jan Brüdigam. Robotics and AI Institute (RAI), Zurich. arXiv 2610.36270, submitted October 26, 2026.
Paper: https://arxiv.org/abs/2610.36270. Code status: the authors do not release a complete implementation of phase-conditioned PPO or the G1 training stack. The Spot work builds on the public ReLIC infrastructure, which is a useful reference for the hierarchical controller but not an implementation of the paper's high-level phase-learning method.
One-Sentence Summary
The paper trains one high-level loco-manipulation policy with a training-only phase state machine, phase-gated rewards, an actor-side phase-prediction task, a phase-conditioned residual critic, and phase-aware resets, then removes every explicit phase signal at deployment so Spot and Unitree G1 execute long behavior sequences autonomously.
Research Context and Motivation
Whole-body loco-manipulation asks a robot to switch repeatedly between locomotion, posture adjustment, contact formation, lifting, and transport. A tabletop pick-and-place policy can often finish in a short, relatively stable interaction window. A task such as lifting a box from the floor and carrying it forces the same controller to manage balance while crouching, sustained forearm contact, force closure, and walking with a changed center of mass. The longer the sequence becomes, the less reasonable it is to expect a single undifferentiated reward to discover the intended order by trial and error.
The failure mode is not merely slow convergence. A standard PPO agent may find a local behavior that receives partial reward while violating the actual task. It can kick or drag a box instead of lifting it, or learn the first approach phase and then collapse when the environment begins to produce a different reward and contact regime. These outcomes arise because early interactions shape the replay distribution: later phases are rarely reached, so their values remain poorly estimated and their successful actions receive little opportunity to be reinforced.
One branch of prior work avoids this exploration problem with human motion capture, teleoperation, or large motion datasets. Those data provide strong kinematic priors and have enabled impressive humanoid skills, but they introduce substantial curation cost and often make the learning recipe specific to a human-like embodiment. The paper starts from a different premise: many manipulation tasks already contain an explicit behavioral decomposition, and this decomposition can be used as training structure even when no demonstration is available.
Finite-state machines are the obvious way to express that structure. A conventional system might train one policy per stage and invoke the next policy after a hard-coded transition. That design is attractive for quasi-static manipulation, where the end of one skill can be made fairly stable. It is much less reliable for loco-manipulation, where momentum and contact persist through transitions. A handoff can ask the next controller to begin from a state it never learned to stabilize, and small observation errors can cause repeated switching around the threshold.
The authors therefore preserve the task's phases during training but refuse to make phase selection an external runtime controller. A single actor must learn the complete sequence and manage transitions internally. The central question is how to expose that actor to all stages, keep phase-specific rewards from fighting each other, and ensure that the shared representation retains enough task-progress information after the simulator's phase labels are removed.
The work also targets embodiment independence. Rather than coupling the method to one humanoid, it adopts a common high-level command interface over platform-specific low-level controllers. This creates a clean separation: the low-level policy handles balance and command tracking, while the high-level policy decides where to move, how low to crouch, when to contact an object, and when to carry it. Spot and G1 then test whether the same learning logic survives across quadrupedal and humanoid dynamics.
Problem Formulation and Hierarchical Control
The task is modeled as a partially observable Markov decision process. The tuple contains states, observations, actions, a stochastic transition model, rewards, and a discount factor:
$$\mathcal{M}=(\mathcal{S},\mathcal{O},\mathcal{A},\mathcal{P},\mathcal{R},\gamma)$$
The agent does not receive the full physical state. At time $t$, it observes $\boldsymbol{o}_{t}$, samples an action from $\pi(\boldsymbol{a}_{t}\mid\boldsymbol{o}_{t})$, and produces a trajectory of alternating states and actions:
$$\tau=(\boldsymbol{s}_{0},\boldsymbol{a}_{0},\boldsymbol{s}_{1},\boldsymbol{a}_{1},\ldots)$$
Learning seeks a policy that maximizes expected discounted return:
$$J(\pi)=\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}\mathcal{R}(\boldsymbol{s}_{t},\boldsymbol{a}_{t})\right]$$
This formalism matters because the paper's structural priors are not additional inputs at test time. They alter the training distribution and the value-learning problem inside the same POMDP. The actor still has to infer what to do from observations once deployment begins.
The action space is reduced through a two-level controller. A low-level whole-body policy tracks commands related to motion and posture, while a high-level policy produces those commands and directly manages the arm targets needed for manipulation. For G1, the low-level controller regulates 15 leg and waist degrees of freedom. It observes the arms but does not control them. This asymmetry is important for box lifting: the high-level policy can shape the arm enclosure, while the low-level controller concentrates on maintaining a stable base.
The low-level command is five-dimensional:
$$\mathbf{c}_{t}=[v_{x},v_{y},\omega_{z},z_{\mathrm{base}},\theta_{\mathrm{pitch}}]^{T}$$
The components specify planar velocities, yaw rate, pelvis height, and torso pitch. The G1 controller is a three-layer MLP with hidden widths 512, 256, and 128, and its observation vector has 104 dimensions. The observation groups include pelvis and torso inertial measurements, velocity and posture commands, 14 arm-target joint values, 58 joint-position and velocity values, and the previous 15-dimensional low-level action.
High-level actions are not absolute command vectors. They are residual changes applied to the previous command and clipped to physical bounds:
$$\mathbf{c}_{t}=\operatorname{clip}\left(\mathbf{c}_{t-1}+\mathbf{a}_{t}\Delta_{\mathrm{scale}},\mathbf{c}_{\min},\mathbf{c}_{\max}\right)$$
The recurrence turns smooth command evolution into a property of the action parameterization. It is especially useful for crouching and standing, where a large jump in pelvis-height or pitch targets can destabilize the base before the low-level tracker has time to respond.
G1 training also constrains the feasible command region. Inverse-kinematics keyframes provide valid examples across different heights and torso pitches, and the convex hull of admissible standing keyframes defines the allowed region. For moving commands, the feasible envelope contracts as target velocity rises. This restriction prevents the high-level learner from spending its budget on posture commands the body cannot physically track, while also placing crouching postures close enough to the nominal standing state for exploration to find them.
Method: Making Phase Structure Useful Without Making It a Runtime Crutch
A state machine defines behavioral windows
The environment maintains a discrete phase variable $\psi_{k}$. A transition uses local success conditions such as distance to an object, object height, or contact status. These conditions are not time schedules: episodes can take different amounts of time and still advance through the same behavioral order. Simulation-only quantities are allowed to participate because the state machine exists only in training.
The transition logic includes hysteresis. Progress requires a strict condition, while regression is intentionally easier. The agent can therefore remain in the current phase long enough to explore a difficult behavior instead of bouncing between adjacent labels when a noisy contact measurement crosses the threshold. The definition permits limited overlap between phase states, but the practical benefit is a more stable curriculum and a clearer local reward signal.
This design differs from chaining separately trained sub-policies. There is one actor, one shared feature trunk, and one deployment path. The phase state machine controls what the learner sees and optimizes during training, not which controller is called at execution. Once training ends, no phase index, privileged contact value, or simulator state machine is passed to the actor.
Phase-gated rewards separate incompatible objectives
A long task usually contains reward terms that should not be active simultaneously. In an approach phase, closing distance to the box is useful. In the following crouch phase, continuing to maximize approach speed can cause a collision, while the intended behavior is to regulate base height and maintain balance. Clamp, lift, carry, and offload require yet another balance among contact force, object height, locomotion, and stability.
The paper defines a phase-gated reward in which only the terms associated with the active phase contribute to the task reward. It also adds a small positive incentive for moving forward through the required sequence and a larger penalty for moving backward. The idea can be expressed as a sum over phase-local rewards, a forward bonus, and a backward penalty:
$$\begin{aligned}r_{t}={}&\sum_{k=1}^{N}\mathbf{1}[\psi_{t}=k]\,r_{k}(\boldsymbol{s}_{t},\boldsymbol{a}_{t})\\&+\lambda_{+}r_{\mathrm{forward}}-\lambda_{-}r_{\mathrm{backward}}\end{aligned}$$
The exact reward weights are task engineering choices, but the control principle is explicit. Gating removes gradients from objectives that are not yet relevant. Progression shaping discourages oscillation across phase boundaries and prevents an agent from collecting local rewards by repeatedly entering and leaving a phase. The sequence prior improves exploration while leaving the actor responsible for actually completing each transition.
An auxiliary classifier forces task progress into actor features
The actor does not observe $\psi_{k}$, so the policy cannot simply memorize a phase-conditioned input. A second head is attached to the shared actor trunk and trained to predict the simulator's phase label. For a sample belonging to phase $k$, with predicted class probability $\hat{p}_{k}$, the auxiliary loss is cross entropy:
$$\mathcal{L}_{\mathrm{CE}}=-\sum_{k=1}^{N}\mathbf{1}[\psi_{t}=k]\log\hat{p}_{k}(\boldsymbol{o}^{H}_{t})$$
This loss does not become part of deployment inference. It shapes the latent features used by the action head, encouraging them to encode whether the robot is still approaching, has begun crouching, has established a clamp, or is carrying the object. The last section of Figure 3 shows the same distinction: phase prediction sits beside the actor's action path, while deployment retains only the observation-to-action mapping after training support is removed.
The ablation makes the representation argument unusually concrete. On Spot tire rolling, success falls from $89\%\pm 2\%$ to $44\%\pm 9\%$ when phase prediction is removed. On G1 two-box pushing, it falls from $92\%\pm 2\%$ to $14\%\pm 6\%$. Both variants still receive the same environment and reward-gating setup, so the large gap isolates the benefit of the auxiliary supervision and its effect on shared features.
A residual critic shares global value while specializing locally
The critic receives the state and a one-hot phase encoding. It does not create a completely separate value network for every phase. A shared trunk produces a global base value, while phase-specific residual heads estimate local corrections:
$$V(\boldsymbol{s}_{t},\boldsymbol{e}_{k})=V_{\mathrm{base}}(\boldsymbol{s}_{t})+\Delta V_{k}(\boldsymbol{s}_{t})$$
The base head sees the long-horizon task as a whole. The residual head can account for a local preference, such as reducing base height during crouching or preserving a stable enclosure during hold. Because the correction is additive, the value estimate can change across a phase boundary, but the shared component keeps every phase connected to the same overall objective.
The authors regularize the residual parameters to keep the base head responsible for most learning:
$$\mathcal{L}_{\mathrm{reg}}=\lVert\theta_{\mathrm{res}}\rVert_{2}$$
A plain MLP may be pulled by conflicting gradients from very different phase rewards. Fully independent heads avoid interference, but they can introduce an unbounded jump at a transition and lose the global relationship between phases. The residual design is a middle position. It offers local capacity and boundary correction while preserving a shared interpretation of task progress.
Phase-aware resets broaden the exploration distribution
Starting every episode in phase one makes later learning causally inaccessible. If the actor has not learned the first push, it will not experience the second box, and the critic will receive almost no informative samples from later contact dynamics. The paper therefore samples episodes at randomized phases early in training. For a tire, this may mean starting near an already upright tire. For box lifting, it can mean starting with the box held in the air.
The reset curriculum gradually reduces mid-task initialization and eventually returns all starting states to the first phase. This is not a deployment shortcut. It changes the distribution from which training trajectories are collected so that all behavioral windows can be optimized in parallel. The authors borrow the motivation from DeepMimic, where later parts of a motion are similarly exposed before the full sequence is mastered.
The box-lifting tasks require curated keyframes rather than arbitrary state perturbation. Reliable bimanual force closure is difficult to discover from the ground state, and a useful keyframe must place both arms around the object without penetrating it. The authors use these keyframes as reset examples, then anneal them away. This improves learning, but it also introduces a manual dependency that later limits scalability when the object geometry changes.
Figure 3 gives the complete architecture. The reset curriculum supplies the simulator with diverse phase starts. A state machine emits the phase, which gates rewards and conditions the critic. The actor shares one trunk between residual command generation and phase prediction. The low-level policy tracks the resulting high-level commands. At deployment, the phase machine, critic, privileged state, and auxiliary target disappear.
flowchart LR
R["Multi-phase reset curriculum"] --> E["Simulator"]
E --> S["Environment state machine"]
S --> P["Active phase"]
P --> G["Phase-gated rewards"]
P --> C["Phase-conditioned critic"]
O["High-level observation"] --> A["Shared actor trunk"]
A --> D["Delta command head"]
A --> H["Auxiliary phase head"]
H --> X["Cross-entropy loss"]
D --> L["Low-level whole-body policy"]
L --> E
C --> U["PPO update"]
G --> U
X --> U
U --> A
D --> T["Deployment actor path"]
The diagram separates the two execution graphs that are easy to confuse. Training includes the simulator state machine and privileged phase conditioning. Deployment contains only the high-level observation, shared actor, delta action, and low-level tracking controller. This is the paper's main conceptual move: phase information can be a temporary training scaffold without becoming a permanent runtime dependency.
Tasks, Platforms, and Hardware Deployment
The experiments cover five tasks and two platforms. Spot rolls a tire through four behavior phases. G1 performs single-box and two-box pushing, ground-level lifting, and lift-carry. The G1 tasks range from three to seven phases. The longer tasks require a full transition from standing, to crouching, to forearm-based clamping, to lifting, and then to walking while maintaining the object.
| Task | Platform | Phases | Behavioral sequence |
|---|---|---|---|
| spot_tire_rolling | Spot with arm | 4 | Approach, upright, roll, tilt-down |
| g1_push_box | Unitree G1 | 3 | Approach, push, rise |
| g1_push_two_boxes | Unitree G1 | 5 | Approach 1, push 1, approach 2, push 2, rise |
| g1_lift_box | Unitree G1 | 5 | Approach, crouch, clamp, lift, hold |
| g1_lift_carry_box | Unitree G1 | 7 | Approach, crouch, clamp, lift, carry, offload, rise |
All policies are trained in the GPU-accelerated mjlab environment. G1 policies are exported through exploy into ONNX and deployed with a custom ROS2 stack. Spot uses a separate control framework. Object mass, size, and friction are randomized, and the main hardware experiments use external motion capture to track the robot and object.
Figure 2 shows the low-level G1 controller and its feasible command envelope. Training combines arm trajectories, inverse-kinematics keyframes, and command interpolation. The feasible-region plot makes the pitch and height bounds visible: standing has a larger admissible region, while crouched postures and locomotion shrink or reshape the safe command set. The hardware timeline on the right shows the same controller producing upright walking, deep crouching, crouch-walking, and arm-control states.
The G1 lift is more demanding than a kinematic sequence alone would suggest. Contact runs along the forearms rather than only at the end effectors. This creates an under-constrained contact problem: small changes in arm pose can alter the normal-force distribution, object slip, and the load transmitted back into the torso. The low-level controller must preserve balance while the high-level controller maintains an enclosure around the box, and the whole system must remain stable long enough to lift and move.
The authors also demonstrate motion-capture-free execution for G1 push and lift. The high-level policy uses object states expressed in the robot's local frame, so external tracking is not logically required. Onboard state estimation supplies the robot pose, and an ArUco marker gives the object-relative estimate. Figure 4 shows the state-estimation view, the first-person marker detection, and the robot holding the box.
This onboard experiment is evidence for a modular perception-to-control boundary, not a general visual-robustness claim. The policy was trained with ordinary observation noise but not with explicit marker dropouts. The authors describe the execution as zero-shot robust to this setup, while the use of a fiducial marker leaves open how performance changes under occlusion, reflection, or loss of visual detection.
Sim-to-Real Changes for Sustained Bimanual Contact
Initial G1 hardware trials exposed arm oscillation and dropped boxes. The simulation's immediate actuation assumption did not match the actual high-frequency response of the arms. The authors randomize actuator delay by up to five physics steps, which is 25 ms at 200 Hz. This change lets the policy learn corrective actions that account for lag instead of relying on unrealistically synchronized feedback.
MuJoCo's soft contact model creates a second mismatch. Long holds produce small contact slips, and MuJoCo Warp does not provide a native correction that fully removes them. The paper uses a friction curriculum that begins with a high sliding coefficient, up to 3.0, to help the policy discover a stable grasp. It then anneals toward 1.5 to 2.0. The final values are still elevated relative to the physical interface, so the schedule is best understood as a learnable transition rather than a complete identification of contact physics.
The third adaptation addresses force rather than kinematics. The simulated combination of incompressible contact and micro-slip encouraged excessive gripping commands, which caused shoulder motors to overheat on hardware. A reward penalty was considered unreliable, so the authors clamp the allowable difference between commanded and current joint positions. The clamp behaves as an implicit torque limit. A 7 N·m equivalent offset, computed from the position gain, provided a balance between secure clamping and thermal safety. With these changes, the hardware maintained the grasp for up to five minutes.
| Hardware issue | Training or control response | Reported setting | Observed purpose |
|---|---|---|---|
| Arm oscillation and dropped box | Randomized actuator delay | Up to 5 steps, 25 ms at 200 Hz | Compensate for latency and smooth the grasp |
| Contact micro-slip in MuJoCo | Friction curriculum | Start as high as 3.0, anneal to 1.5 to 2.0 | Make force closure learnable before reducing the sim-to-real mismatch |
| Overheating shoulder actuators | Clamp commanded-minus-current joint offset | Equivalent to 7 N·m through $K_{p}$ | Act as an implicit force and thermal limit |
| Long-duration holding | Combine delay, friction, and force controls | Steady holds demonstrated for up to 5 minutes | Preserve contact while avoiding unsafe motor load |
These interventions are diagnostically useful because they attack different layers of the sim-to-real gap. Delay randomization changes the policy's temporal response. The friction curriculum changes the contact regime seen during learning. The position-offset clamp changes the safe action set shared by simulation and hardware. None of them is presented as a universal replacement for accurate contact modeling, and the paper explicitly notes that the final simulated friction remains high.
Ablations: Which Parts Carry the Learning Problem?
The quantitative ablation evaluates Spot tire rolling and G1 two-box pushing with three random seeds. The G1 task is selected because its five phases expose policy collapse without making reward tuning as sensitive as in box carrying. The comparison includes the complete method, a variant without phase-prediction supervision, a run initialized only in the first phase, a standard MLP critic, and fully decoupled phase heads.
| Task | Full method | No phase prediction | Initial-phase reset only | Seeds |
|---|---|---|---|---|
| Spot tire rolling | 89% ± 2% | 44% ± 9% | Can succeed, with higher variance | 3 |
| G1 two-box pushing | 92% ± 2% | 14% ± 6% | Complete collapse on the complex sequence | 3 |
The phase-prediction ablation is the strongest result. The no-prediction variant still has the same actor observations and receives the same gated rewards, yet it loses most of the G1 success rate. This supports the specific representation hypothesis: local rewards alone do not ensure that the shared trunk encodes the behavior boundary. In a seven-phase or highly contact-sensitive task, that missing boundary can make the same observation region require incompatible actions without the network preserving a useful estimate of progress.
The reset ablation separates exploration from architecture. On Spot, starting only in phase one can still produce success, but the training curve is more variable. On G1, mastering the first box does not transfer cleanly to the distinct rewards and dynamics of the second. Because the first-stage-only run never exposes the critic to representative later states, it cannot form a reliable value estimate for them. Multi-phase initialization gives all behavioral windows a chance to contribute gradients before the curriculum returns to the real start distribution.
The critic topology comparison is more nuanced. The residual value network has higher success and lower variance during training, but the authors report that the gap on standard benchmarks is relatively subtle. Its stronger argument comes from the complex carrying behavior and from the shape of the function it induces: a shared base value plus bounded local corrections is less prone to conflicting gradients than a plain MLP and less discontinuous than fully independent heads.
Figure 5(c) of the ablation study analyzes 1,000 successful Spot episodes. Because phase durations vary, the trajectories are anchored at phase transitions and linearly warped to a common time axis. The predicted phase closely follows the ground truth inside each window, with most error near transition boundaries. That is exactly what the method needs: the auxiliary head does not have to act as an oracle during deployment, but the features should retain enough temporal structure to produce a coherent action sequence.
A careful reading should not turn the 89/44 and 92/14 comparisons into universal performance claims. They are simulation ablations on two tasks and three seeds. Their value is causal evidence for the design choices, while the hardware experiments establish feasibility on real systems. The paper does not report a large hardware success-rate matrix across object dimensions, lighting conditions, contact materials, or repeated autonomous runs.
Public Code and Reproduction Boundary
The paper does not provide a standalone phase-conditioned PPO repository. The closest public dependency is ReLIC, which supplies the hierarchical loco-manipulation architecture and the Spot implementation used in this work. At commit 27f8033c5064d32f049a17accb71cd1091422878, spot_joint_actions.py shows how arm, leg, and whole-body joint targets are written into the simulation action interface. interlimb_env_cfg.py contains the velocity, arm, leg, posture, and proprioceptive observation groups used by the command interface.
The low-level G1 MLP dimensions also align with the public ReLIC PPO configuration. In rsl_rl_cfg.py, actor and critic hidden layers are set to [512, 256, 128], action clipping is 0.2, the value-loss coefficient is 1.0, and the learning rate starts at 0.001 with adaptive scheduling. These details make the low-level infrastructure inspectable, but they do not implement the paper's phase state machine, gated task rewards, auxiliary actor classifier, residual phase heads, or multi-phase reset curriculum.
The G1 training environment, low-level checkpoint, hardware deployment stack, and task-specific code are not released. A reproduction therefore cannot be reduced to cloning ReLIC. A team would first need to reproduce a stable command-tracking controller for the target body, then implement the high-level phase machinery on top of the same interface. The paper's architecture is modular enough for that work, but the key engineering effort is in command feasibility, contact simulation, reset construction, and hardware safety rather than in a single network definition.
Limitations and Risks
The first limitation is stated by the authors. Increasing the number of phases and extending the horizon strains representational capacity and value estimation. A large set of local residual heads can become difficult to coordinate, while the global value component may dominate the fine-grained changes needed inside a phase. The experiments reach seven phases, so the result does not establish that the same recipe scales to the dozens of interleaved stages common in household or warehouse workflows.
The second limitation is manual task engineering. The designer must decide how to segment behavior, define each transition condition, select reward terms, tune their weights, and construct reset keyframes. For box carrying, changing the box shape can require new keyframes that preserve a valid bimanual enclosure. This is a real scalability cost and one of the main reasons the authors propose automatic phase discovery as future work.
The third limitation concerns contact simulation. Delay randomization, elevated friction, and an implicit torque limit make the G1 transfer possible, but they are calibrated to the experiments. The final friction still differs from the real contact, and the safe joint-offset limit depends on actuator gains and thermal properties. A different forearm geometry, box material, payload, or controller gain would require renewed validation.
There is also an evidence gap in the hardware evaluation. The paper shows successful qualitative deployments, and the five-minute hold is a meaningful stress result. It does not provide per-object success rates with confidence intervals, repeated trials for every task, or a systematic failure breakdown. The simulation and hardware results answer different questions: one supports component attribution, while the other demonstrates that the complete stack can run on physical robots.
Finally, the ArUco-based evaluation reduces dependence on external motion capture but introduces a perception assumption. A marker-based object estimate is useful for isolating the control contribution, yet it is not a general manipulation perception system. Occlusion, reflective surfaces, camera calibration drift, and marker absence remain outside the reported evidence. The paper's zero-shot claim should therefore be read as zero-shot from simulation to this hardware and perception setup, not zero-shot across arbitrary visual conditions.
Conclusion and Outlook
The contribution is a way to place task structure where it is useful during learning without leaving that structure in the deployed policy. Phase-gated rewards localize the optimization signal, multi-phase resets expose the learner to every behavior window, the auxiliary classifier preserves task progress in shared actor features, and the residual critic separates global return from local phase effects. The actor remains a single observation-to-action policy at test time.
Spot tire rolling and the G1 lift-carry sequence provide complementary evidence. Spot tests the architecture on a non-humanoid platform with an arm. G1 tests the harder combination of crouching, sustained bimanual contact, lifting, locomotion, and onboard execution. Across the ablations, the phase-prediction and reset components produce the largest behavioral changes, which points to representation and exploration distribution as central bottlenecks in long-horizon locomotion and manipulation.
The authors identify three natural next steps. Phase guidance could be moved to off-policy learning to improve sample efficiency. Phase structure could be discovered from offline data or sample-based model predictive control instead of being drawn by hand. The framework could also be extended to multi-robot reinforcement learning, where phases become coordination contracts as well as single-robot behavior windows. The present paper establishes the single-policy recipe; these directions determine how far its manually specified structure can scale.
Key Takeaway
Long-horizon task phases should shape the actor during training, not remain as an external controller that makes decisions for the actor at test time.
SOURCE LINKS
