From Video to Vision: Learning Generalizable Humanoid Carrying from Human Motion

The teacher, assistant, and student curriculum

Stage one models carrying as a partially observable Markov decision process and trains with PPO. The action is the full set of joint angles. The observation includes robot proprioception, object state, and the reference trajectory. An asymmetric actor-critic lets the critic access reference body position and orientation while the actor omits anchor linear velocity, because the teacher is intended to operate in a motion-capture room.

The reward has five families: body tracking, object tracking, guided contact, regularization penalties, and foot-sliding penalties. Tracking terms use a Gaussian-shaped error:

$$ r(\bar e_s,\sigma_s) = \exp\left(-\frac{\bar e_s}{\sigma_s^2}\right). $$

The authors track key body parts such as feet, legs, and upper arms rather than every segment. Orientation receives a higher weight than position to discourage poses that match a point while twisting the body into an implausible configuration. Training always samples from the first frame because starting halfway through a carrying motion does not guarantee that the robot is already holding the object correctly.

Stage two keeps the policy and observations but changes the objective. Global anchor-position tracking and the robot's own body-position tracking are removed. Body orientation remains, while object-position and contact rewards are increased. A joint-limit termination condition prevents the assistant from exploiting the relaxed objective with unsafe extreme poses.

This stage is the source of spatial generalization. At zero offset, the teacher already performs well. As the object moves farther from the demonstrated location, the assistant can choose a new path and body placement while continuing to track the object and maintain contact.

Stage three distills the assistant into a deployable student with DAgger-PPO. The student loses global proprioceptive position and reference joint positions. Its inputs retain a reference anchor-orientation error, joint position, joint velocity, base angular velocity, previous actions, object relative pose, and contact-point positions. A history length of five compensates for short-term observation noise.

DAgger supplies corrective supervision on states visited by the learner, while PPO still optimizes the task. The paper reports 50,000 iterations for this stage. Imitation runs during the first 3,000 steps, after which the PPO loss is introduced. The imitation coefficient is 1.0 and the PPO coefficient is 0.1.

flowchart LR
    V["100 human videos"] --> G["GENMO motion recovery"]
    G --> R["GMR retarget to 29-DoF Tiangong 3.0"]
    R --> A["Object and contact augmentation"]
    A --> D["4000 reference trajectories"]
    D --> T["Stage 1 teacher: full tracking"]
    T --> S["Stage 2 assistant: relax body position"]
    S --> P["Stage 3 DAgger-PPO student"]
    P --> Q["QuatNet and RGB-D relative object pose"]
    Q --> H["Mocap-free physical deployment"]
    

The curriculum is monotonic in the amount of task context available. Each stage begins from a policy that already solves the preceding problem, and each reduction in privileged information is paired with a mechanism that preserves the missing guidance.

QuatNet and the visuomotor deployment loop

Removing global position also removes a strong cue for heading toward the destination. QuatNet predicts the reference anchor orientation from a 39-dimensional input containing object relative pose, joint positions, and target position. Two long short-term memory layers produce a four-dimensional quaternion.

The network is trained on 500 motion sequences, approximately one million frames, with 20 percent held out for validation. The target is the anchor quaternion from the reference motion, and the supervised loss is

$$ \mathcal{L}_{\mathrm{quat}} = \left\| \mathbf{q}_{\mathrm{pred}} - \mathbf{q}_{\mathrm{gt}} \right\|_2^2. $$

Training runs for 5,000 epochs. The paper records training and validation loss, rotation-angle error, and quaternion difference. The curves show convergence, but the downstream experiments are more informative than the loss alone.

At deployment, a head-mounted RGB-D camera detects a fiducial tag attached to the object. Let $P$, $C$, $M$, and $O$ denote the pelvis, camera, marker, and object frames. The object pose in the pelvis frame is

$$ {}^{P}\mathbf{T}_{O}(t) = {}^{P}\mathbf{T}_{C}(\mathbf{q}_t) {}^{C}\mathbf{T}_{M}(t) {}^{M}\mathbf{T}_{O}. $$

The camera-to-pelvis transform depends on the head joint configuration and calibrated extrinsics. The marker-to-object transform remains fixed after tag attachment. Validation, temporal smoothing, recent observations, and forward kinematics from the palms handle short occlusions. If the tag is lost for a long period before grasping, the robot waits rather than advancing with an unchecked object estimate.

Simulation results and what they show

Training uses Isaac Lab, and simulation evaluation uses MuJoCo. The task contains four object placements and two shape-size configurations. A trial succeeds when the object ends within 0.3 meters of the target. The reported metrics are carrying success rate, grasping success rate, and final position error.

The following table selects size configuration one at all four positions. Values are carrying success in percent and error in meters.

The assistant is the strongest policy under displaced objects. It reaches 90.4, 88.1, and 86.9 percent at positions two through four, with errors below ten centimeters. It uses global position, so this result is not a claim of mocap-free deployment. It identifies which policy the student should emulate once the global inputs are removed.

The student generalizes unevenly. At the nominal position it exceeds the assistant, reaching 94.3 percent without QuatNet and 94.9 percent with it. At larger offsets, its carrying rate falls to 35.2, 58.4, and 40.2 percent without QuatNet. Grasping remains close to perfect. The robot can usually acquire the object, but estimating the destination becomes harder without global proprioception.

The staged ablation exposes the same tradeoff. Each cell below is carrying success, grasping success, and error:

At the zero-offset position, teacher and assistant are nearly tied. The fourth position reveals the effect of changing the reward: carrying success rises from 68.6 to 89.5 percent and error falls from 0.345 to 0.071 meters. The assistant is not simply better at tracking; it is better at abandoning the demonstrated route when the object requires another approach.

QuatNet does not recover all of the omitted position information. It improves over the plain student at the nominal position, but the displaced-object results remain lower than the assistant. The authors attribute this to the indirect nature of predicted heading. A reference trajectory supplies a continuous direction over time, while a quaternion prediction can accumulate error as the robot transports the object.

Physical tests compare the assistant with motion capture against the vision student in a normal room. The sample is ten trials for each of three positions.

The vision policy succeeds one fewer time at each displaced position and has a larger final error. This is useful evidence that the observation reduction survives the sim-to-real gap, but it is not a precise estimate of field reliability. The object also carries a fiducial tag, and the test uses one physical humanoid.

Why the curriculum matters

Object augmentation alone would create references that the teacher can track, but it would not force the policy to stop using global body position. Reward relaxation alone would ask the policy to generalize to object configurations that the data may not contain. Distillation alone would train a student on states that the privileged assistant never needed to visit. MOVER couples all three so that each change prepares the next one.

The deployment interface is also narrower than a general perception system. QuatNet supplies a reference heading, and the camera pipeline estimates object pose through a known tag. The system still needs a target location and assumes the marker-to-object transform is calibrated. Transparent, reflective, deformable, or heavily occluded objects would require a different object-modeling and pose-estimation approach.

The paper's strongest result is not a single success percentage. It is the demonstration that deleting privileged observations does not have to happen as an abrupt deployment patch. Training can deliberately reduce privileged input while preserving the task-relevant relative structure.

The most important unresolved question is how far this recipe scales beyond the demonstrated task. A larger video corpus may teach different carrying styles and object interactions, but it will also expose the limits of a single reference-motion space and tag-based pose estimation. The reward and observation design would need to accommodate obstacles, human proximity, deformable objects, and longer-horizon navigation.

Limitations

The authors state two limitations directly. First, the amount of training data is insufficient for broad generalization over object shapes and motion patterns. Second, removing global proprioceptive position causes non-negligible target-position error; the student's displaced-position failures in the tables are consistent with this explanation.

Three further risks follow from the experimental design. The perception system relies on a fiducial marker, so it does not establish performance on unmarked everyday objects. The real-world evaluation uses ten trials per position, with no confidence interval or cross-robot replication, which makes small differences difficult to interpret. The paper also provides limited stress testing of simulator mismatch, sensing latency, and contact variation.

The baseline comparison needs careful reading. HumanX records zero success in this setup, which demonstrates incompatibility with the task configuration but does not explain the failure mode. OmniRetarget degrades sharply as the object moves, supporting the paper's argument about reference dependence, yet the table alone cannot separate retargeting quality from policy design. These qualifications do not overturn the main result, but they limit how broadly it should be generalized.

Conclusion

MOVER turns human carrying videos into contact-consistent humanoid references, expands them across object configurations, and trains a policy that can eventually operate without global body position. The teacher establishes control, the assistant learns to depart from the demonstration route, and DAgger-PPO transfers the behavior to a student with a deployable observation set. QuatNet and RGB-D sensing supply the remaining relational guidance.

The paper's main systems lesson is that observation removal should be part of training, not an afterthought in deployment. Privileged information can bootstrap control, but the curriculum must deliberately create states in which the policy uses the sensors available on the robot.

Future work should replace the fiducial marker with robust object perception, expand the diversity of motions and objects, improve target-distance estimation without global proprioception, and evaluate several humanoids with statistically meaningful real-world trials. Those steps would test whether MOVER is a general method for video-to-humanoid transfer rather than a strong demonstration under a carefully instrumented carrying setup.

Train with privileged state, then earn generalization by removing it one controlled step at a time.