METATRON: A Promptable Whole-Body Execution Layer for Humanoids

One-sentence summary: METATRON trains one generative humanoid policy to complete sparse spatial, velocity, style, and scene prompts, then moves task-specific logic into ordinary stateful programs so that a frozen Unitree G1 controller can walk, recover, accept hand targets, sit on furniture, and support compliant interaction without task-by-task retraining.

Figure 1: The paper teaser. One frozen G1 execution policy covers walking, recovery, teleoperated object pickup, and furniture interaction through different sparse prompts.

1. Research context: the missing layer is a reusable motor substrate

A humanoid system that appears to perform one continuous household task may internally switch among several controllers. Locomotion takes the robot to a table, manipulation moves the hand and maintains contact, balance recovery reacts to an unexpected push, and physical interaction with a person requires compliance. Every controller repeats much of the same reasoning about support polygons, center-of-mass transfer, contact, and whole-body coordination. The engineering burden is therefore not only the quality of each individual policy. It is also the fragility created by handoffs between policies and the pressure to retrain a network whenever the task changes.

METATRON separates task intent from motor execution. A task program decides what should happen, while a pretrained and frozen policy determines how the body should coordinate to make it happen. If the execution policy is sufficiently broad, a developer should be able to add a new behavior by writing code that emits approximate targets rather than by defining a new reward, collecting a task-specific dataset, and training another neural controller. This distinction is the central architectural claim of the paper. It converts part of humanoid behavior development into software development: rules can be reviewed, tests can be repeated, and a coding agent can iterate on a program without changing the policy weights.

The method inherits the masked motion completion idea behind MaskedMimic. During training, the model may see a complete motion. During deployment, it receives only fragments: future positions of selected body parts, a velocity direction, a handful of style frames, or the geometry of an object in the scene. The missing trajectory is not an error to be interpolated mechanically. It is a degree of freedom that the policy resolves while preserving balance and contact. This differs from conventional trajectory tracking, where a reference generator is expected to provide a sufficiently dense and dynamically plausible motion. METATRON treats that dense motion as optional context and asks the policy to synthesize the missing coordination.

The second motivation is data coverage. Motion-capture corpora include walking, running, dancing, sports, and acrobatics, but they do not systematically contain the awkward configurations a robot reaches after falling. Scene-interaction datasets contain useful behavior such as sitting and lying on furniture, yet naive human-to-robot retargeting can introduce penetration and contact errors. Recovery trajectories generated with reinforcement learning can fill the gaps near failed states, while scene-aware retargeting supplies body-object relationships that are absent from ordinary mocap. METATRON therefore trains a single student from several groups of experts rather than pretending that one source of motion data covers all physical situations.

The third motivation is the use of coding agents. The first version of a skill is usually easy to express: place the torso above a seat, move the feet to named support points, or command forward velocity with a desired heading. The difficult work appears when the policy hesitates, shakes, collides, or chooses an unintended gait. The paper uses coding agents as program engineers. They inspect motion clips, construct simulation tests, fit compact periodic rules, and revise the skill code after observing failures. The agent therefore optimizes program parameters and control logic, not the base policy. A successful iteration improves the interface to an existing motor capability rather than adding another task-specific network.

The scope is deliberately bounded. METATRON does not claim to solve open-world perception, navigation, or multi-minute task planning. It assumes that another component can supply a useful seat bounding box, a target for the hand, the intention behind a handshake, or the approximate location of an object. The policy turns such signals into motion. It does not discover the object, localize itself over a full building, or construct a long task tree. This division is not a minor implementation detail. It explains both the strength of the system and the conditions under which its experimental evidence should be interpreted.

2. Technical background: masked completion, conditional VAEs, and frozen teachers

Masked motion completion treats a trajectory as a structured tensor with missing entries. One training sample might reveal the future position of the torso but not the hands; another might reveal foot velocities but not the final support configuration. The model learns to infer the missing data from the visible constraints and the robot's proprioceptive history. At deployment, the same mechanism turns sparse user targets into a distribution of plausible completions. A mask is therefore as important as a numeric target. An explicit zero and an unspecified dimension have different meanings, and the network must be able to distinguish them.

A conditional variational autoencoder represents the fact that a prompt can admit several valid motions. Direct regression tends to average incompatible solutions: the model may produce a pose between a left step and a right step instead of choosing either. METATRON instead uses two distributions. The prior sees only information available at deployment and predicts a latent distribution. A privileged posterior sees the full reference motion and simulator state during training, giving the encoder a sharper explanation of the target. A decoder maps the sampled latent together with the robot state to joint targets. The posterior is a training-time teacher, not part of the deployed controller.

The dense experts are trained with reinforcement learning. Each expert observes privileged simulator state and a complete future reference, and it learns to track that reference accurately. The student then imitates the expert's actions while learning a compact latent interface. Once distillation is complete, the simulator state, the full future motion, and the expert are no longer needed. The deployed controller receives robot sensors, the sparse prompt, and the previous action. Optional localization or scene geometry can be included when the skill has it, but motion-capture ground truth is not required.

The action interface follows a BeyondMimic-style offset representation. For the G1 character, the policy outputs commands for 29 joints at 50 Hz. Physics runs at 200 Hz, so each control action is held for four simulation steps. The SOMA character uses 66 actions, with physics at 120 Hz and control at 30 Hz. Joint order, action scale, default pose, PD gains, and observation layout are part of the deployment contract. A model that performs well in simulation is not useful on hardware if any part of this contract is changed silently.

3. The base policy: from dense tracking teachers to a sparse-prompt student

3.1 Dense trackers first learn coordinated motion

The first training stage builds dense motion trackers. A teacher receives privileged simulator state $s_t^{\mathrm{trk}}$, a complete future reference $y_t$, and the previous action $a_{t-1}$, and outputs a high-fidelity joint action:

$$ a_t^{\mathrm{trk}}=\pi^{\mathrm{trk}}\left(s_t^{\mathrm{trk}},y_t,a_{t-1}\right). $$

The word dense does not mean that the teacher sees only the current frame. The reference contains several future timestamps, including desired body positions, orientations, and velocities. The tracking reward sums exponential penalties for multiple error channels while regularizing action change and adding task-specific terms:

$$ r_t=\sum_k w_k\exp(-\lambda_k e_{k,t})-w_a\mathcal{R}_{\mathrm{act},t}+r_t^{\mathrm{extra}}. $$

Here $e_{k,t}$ may represent position, orientation, or velocity error for a selected body, while $w_k$ and $\lambda_k$ determine the relative importance and sensitivity of that channel. The action regularizer discourages abrupt changes between neighboring control steps. The extra term can emphasize particular body points or physical constraints. The G1 teacher tracks global targets for the torso, hands, and feet together with joint-space targets for the rest of the body. It is intentionally stronger than a deployable controller because it has access to information that hardware does not.

3.2 A prompt is a collection of masked geometric streams

The runtime prompt is a structured object rather than a sentence:

$$ g_t=\left(\mathcal{C}_t^w,\mathcal{C}_t^\ell,u_t,\xi_t,b_t\right). $$

The world-frame constraint stream $\mathcal{C}_t^w$ contains body positions or orientations with intended arrival times. The local stream $\mathcal{C}_t^\ell$ contains body poses expressed relative to a target torso position and heading. The planar velocity command $u_t$ is represented in a heading-relative frame. The style term $\xi_t$ supplies a small set of unordered reference frames, and $b_t$ contains oriented bounding boxes for scene objects. Each stream has an independent mask, so a request for a forward velocity does not accidentally imply a hand pose, and a visible zero target is not confused with a missing target.

World-frame targets are useful for statements such as "place the right wrist at the cup in 2.5 seconds." Local targets are better for statements such as "maintain a crouched posture while moving relative to the torso," because they do not require the final global position to be known in advance. Style frames provide qualitative motion cues without imposing temporal alignment. The same crawling examples can therefore be paired with different destinations. Scene objects are reduced to at most two oriented boxes represented by their eight corners, which keeps the network away from raw point clouds or camera images.

The important design property is not that the prompt supports many fields. It is that a task can omit most fields. A skill does not need to synthesize a complete motion clip. It can sketch constraints at a few future horizons, and the policy fills in center-of-mass shifts, foot timing, arm counterbalance, and contact. The paper repeatedly stresses that these constraints are coarse. They are not precise offsets from the current robot state, and they do not encode the full dynamics of the desired action.

3.3 A spherical latent selects among possible completions

Let the deployable observation history and previous action be $x_t=(o_t,a_{t-1})$. The prior and the privileged posterior are modeled as diagonal Gaussians:

$$ p_\theta(\tilde z_t\mid x_t,g_t)=\mathcal{N}\left(\mu_t,\operatorname{diag}(\sigma_t^2)\right), $$ $$ q_\phi(\tilde z_t\mid x_t,g_t,s_t,y_t)=\mathcal{N}\left(\mu_t+\Delta\mu_t,\operatorname{diag}({\sigma_t'}^2)\right). $$

The prior is implemented with a six-layer Transformer that processes the current state, history, world targets, local targets, style frames, and velocity tokens. The posterior uses the complete simulator state and reference only during training. After sampling, the latent is projected onto the unit sphere:

$$ z_t=\frac{\tilde z_t}{\lVert\tilde z_t\rVert_2}. $$

The spherical constraint removes radial magnitude as a trivial degree of freedom. Without it, a generative model could express differences in motion by increasing the norm of the latent while leaving its direction nearly unchanged. Normalization forces the decoder to use the geometry of the representation itself. The resulting vector is concatenated with the current sensor state and previous action before the action decoder predicts joint targets. In the deployed path, the posterior and reference are absent, so there is no direct leakage of privileged simulation data.

3.4 Distillation balances imitation, distribution matching, and temporal stability

The student is evaluated through the posterior path during training. Its decoded action is $a_t^q$, while the dense teacher supplies $a_t^{\mathrm{trk}}$. The objective combines action imitation, KL matching between posterior and prior, unit-norm regularization, dynamics prediction, temporal consistency, and action smoothness:

$$ \begin{aligned} \mathcal{L}={}&\mathbb{E}\left[\ell(a_t^q,a_t^{\mathrm{trk}})+\beta_eD_{\mathrm{KL}}(q_\phi\|p_\theta)\right] +\lambda_\mu\mathbb{E}\left(\lVert\mu_t\rVert_2-1\right)^2\\ &+\lambda_{\mathrm{dyn}}\mathbb{E}\left[\ell\left(F(h_t^q),\operatorname{sg}(d_{t+1})\right)\right] +\lambda_{\mathrm{tc}}\mathbb{E}\left[\ell\left(h_t^q,\operatorname{sg}(h_{t-1}^q)\right)\right]\\ &+\lambda_{\mathrm{act}}\mathbb{E}\left[\ell\left(a_t^q,\operatorname{sg}(a_{t-1}^q)\right)\right]. \end{aligned} $$

The function $\ell$ is a coordinate-wise mean-squared error and $\operatorname{sg}$ stops gradients through the target branch. The KL term pushes the deployable prior toward the better-informed posterior. The dynamics term asks a fused representation to predict the next conditioning signal. Temporal consistency discourages the latent from jumping between nearly identical observations, while the action term reduces high-frequency changes in neighboring commands. For G1, the KL coefficient rises cyclically from $10^{-4}$ to $10^{-2}$ and is annealed repeatedly between epochs 500 and 2000.

3.5 On-policy labels cover states reached by the student

Naive offline distillation learns only from states on the teacher's trajectory. Once the student makes a small mistake, it enters a state that the teacher never supplied, and the error can grow. METATRON labels the current state with the teacher even when the student is controlling the body. During each episode, the posterior student acts with probability 90 percent and the frozen teacher acts with probability 10 percent. The mixture provides clean expert states and also tells the student what a teacher would do after a realistic student error.

A fixed Gaussian noise vector is shared by the prior and posterior within an episode. Only the means and variances respond to the evolving observation. The latent therefore contains episode-level variation while still reacting continuously to the current state. Adjacent timestep supervision excludes pairs that cross an episode reset, because a teleport caused by reset is not evidence that two actions should be continuous.

This training design explains why one student can accept dense pose constraints, isolated body targets, velocity commands, and style examples with very different information content. The expert supplies local action quality. Masking supplies interface diversity. The spherical VAE preserves multiple completions. On-policy labels reduce the distribution shift between training and deployment. None of these mechanisms is sufficient in isolation; the contribution is the combination that makes a broad promptable execution layer trainable.

4. Training data: separate sources for skills, contact, and recovery

The G1 student is distilled from four teacher groups: BONES motion capture, AMASS plus LAFAN, synthetic recovery, and SAMP scene interaction. The approximate group weights are 0.6, 0.2, 0.1, and 0.1. The first two groups provide ordinary locomotion, expressive motion, and dynamic skills. Recovery data covers configurations reached after falling. Scene data adds sitting on furniture, lying on a bed, and other body-object contacts that are not represented well in a conventional mocap library.

Motion retargeting must preserve joint limits, smoothness, and foot contact while aligning the human keypoints to the robot. Scene interaction adds a harder requirement. If body-object contact is broken during retargeting, the robot may penetrate the sofa or distort its pose to avoid penetration. The authors use a two-stage optimization whose second stage includes scene collision cost and joint anchoring. The anchor term biases the solver toward moving the root rather than drastically changing the articulated configuration. This choice preserves the intended relationship between the body and the object.

Furniture is represented by oriented surface point clouds, and internal surfaces are filtered for each motion. The collision term penalizes penetration depth but not contact itself. A person can therefore rest on the seat instead of hovering a few centimeters above it. The paper states that retargeted SAMP data will be released, but the public model package does not include the training motions, recovery clips, robot meshes, or furniture assets. That distinction matters for anyone expecting a complete end-to-end reproduction from the checkpoint alone.

Recovery clips are not created by copying human falling videos onto the robot. They are generated separately for each simulated embodiment with an extended HIL procedure. A collector starts from randomized ground configurations, runs a five-second recovery attempt, and accepts a clip when the final constrained body position error is below 0.25 meters. This filter checks terminal tracking but does not establish sustained balance, contact quality, or hardware safety. The generated motion must still pass simulation evaluation and physical testing before it can be treated as a deployment behavior.

5. The skill layer: ordinary programs sketch targets for a frozen policy

flowchart LR
    A[BONES mocap] --> E1[FSQ PPO expert]
    B[AMASS and LAFAN] --> E2[FSQ PPO expert]
    C[Synthetic recovery] --> E3[Recovery expert]
    D[SAMP scene clips] --> E4[Scene expert]
    E1 --> T[Masked on-policy distillation]
    E2 --> T
    E3 --> T
    E4 --> T
    T --> P[Six-layer prior transformer]
    P --> Z[64D spherical latent]
    Z --> Q[Action decoder]
    Q --> F[Frozen G1 policy]
    S[Skill code and state machines] --> G[Prompt streams and masks]
    G --> P
    F --> R[Joint targets at 50 Hz]

A skill is a stateful function or finite-state machine. At every control step it constructs a prompt and changes the active constraints when its internal state changes. It does not output joint commands and does not need to solve a full trajectory. It specifies a few future body targets: torso position, heading, foot placement, hand target, velocity, style, or scene box. METATRON converts these discrete intentions into balanced motion. The skill remains inspectable code, while the learned policy handles the parts of motor coordination that are difficult to write by hand.

Figure 2: The skill-development workflow from paper Figure 3. The coding agent builds evaluation scenes and revises program rules; it does not retrain the base policy.

5.1 A gait rule turns velocity into five future foot targets

A velocity command by itself can produce high-frequency shuffling. The resulting motion may satisfy the average velocity while wasting energy and remaining difficult to transfer to hardware. The authors fit a compact periodic footprint sketch from motion-capture examples:

$$ \mathbf{p}_i(\tau)=\mathbf{b}_i+ \begin{bmatrix} A(v)\cos\alpha\cos\phi_i\\ W\left(s_i|\sin\alpha|+|\cos\alpha|\sin\alpha\cos\phi_i\right)\\ H\max(0,-\sin\phi_i) \end{bmatrix}. $$

Here $v$ is speed, $\alpha$ is the angle between the travel direction and the robot heading, and $\phi_i=\phi_t+2\pi f(v)\tau+\delta_i$ gives opposite phase for the two feet. The public appendix specifies $A(v)=0.75(0.13+0.065v)$, $f(v)=\min(2,0.83+0.93v)$, $W=0.12$ meters, and $H=0.08$ meters. As speed rises, stride length and cadence both increase. The robot does not attempt to reach a distant point by taking ever faster miniature steps.

The skill emits foot targets at five horizons between 0.1 and 0.5 seconds. The policy then decides how to move the body, load each foot, and maintain balance. Forward motion, lateral motion, and turning are not separate classifier modes. They emerge from the same rule as $\alpha$ changes. The public deployment code adds standing support, wider side steps, and zero-speed handling, showing that a formula which is compact in the paper still needs boundary cases in a real controller.

5.2 Sitting on furniture needs only three coarse targets

Given a seat bounding box, the skill produces a torso target slightly above and behind the seat plus two foot targets outside the front edge at standing width. Every future slot receives the same three static points. The policy decides how to turn, step, lower the center of mass, and place the feet. To stand up, the program raises the torso target along a short forward path and releases the foot gait constraint. The developer describes where the body should end up rather than hand-authoring every frame of a sitting animation.

This example shows the practical benefit of the interface. A new piece of furniture can change the geometry without changing the neural controller. A parameterized rule can be fit from a small sitting benchmark and then evaluated in simulation and on hardware. The limitation is equally clear: the system assumes that perception already provides the seat box and that the robot has been placed near the object. It does not solve visual affordance discovery or navigation to the chair.

5.3 Recovery is a state machine that changes the active prompt

The recovery skill compares torso height with the lowest body point. When the torso is below roughly $0.65H$, the normal task prompt is suspended and an upright posture is requested. When the normalized height rises above $0.90H$, the original task resumes. The two thresholds introduce hysteresis and prevent the state machine from toggling when the robot remains near one boundary. For a simulated character, the recovery prompt can specify a full-body upright pose. On the physical G1, the skill constrains torso height and leaves planar position, heading, and limb coordination to the policy.

The arrival time of the first recovery target depends on the remaining height difference:

$$ \tau_0(h)=\operatorname{clip}\left(\frac{H-h}{v_{\mathrm{rise}}},0.25\,\mathrm{s},1\,\mathrm{s}\right). $$

The value $v_{\mathrm{rise}}=0.8$ meters per second sets a prompt schedule rather than forcing the robot to rise at that exact velocity. Later slots are added every 0.25 seconds and can extend to two seconds. A large height difference gives the policy more time to reorganize the body, while a nearly upright state receives a tighter target horizon. This is a useful example of a skill encoding temporal intent without prescribing the detailed motion.

5.4 Compliance can come from removing constraints

During a handshake or when a person moves the robot's arm, the skill sets velocity to zero and constrains only torso height and heading. It does not specify a hand target. A dense tracking controller would interpret the externally moved arm as an error and try to pull it back. METATRON treats the arm motion as an unconstrained degree of freedom and keeps the body stable. Compliance is therefore not necessarily a separate low-level controller. In this case, changing the prompt and mask changes the control semantics.

That behavior is also a warning against equating sparse prompts with weak control. The absence of a target is meaningful. It tells the policy which dimensions are allowed to vary in response to contact. Designing a safe skill requires understanding this masking behavior, not just choosing a list of positions and velocities.

6. Experiments: broad demonstrations, with quantitative evidence concentrated in two benchmarks

6.1 Hardware and evaluation setup

The G1 policy is trained in Isaac Lab. The deployment path exports the network with TensorRT and uses CUDA graphs on a laptop equipped with an RTX 5090. The measured processing latency from sensor collection to joint-command output is 1.3 milliseconds. This is not the full 20-millisecond control period, and it is not a substitute for measuring deadline jitter, thermal throttling, network delay, or safety response on the robot. The computer communicates with the robot over Ethernet. An Xbox controller supplies velocity and heading commands, while a PICO headset supplies hand targets.

Physical demonstrations use deterministic inference with the prior mean and zero sampling noise. Changing a task changes the prompt observations and masks; it does not change the policy weights or the sensor-processing interface. The hardware demonstrations include forward and backward walking, lateral motion, turning, teleoperated apple pickup, sitting on and standing from a sofa, and recovery from supine, prone, seated, and unusual configurations.

Figure 3: A G1 style-control example from paper Figure 5. Position targets and five unordered crawling poses are combined; the policy supplies temporal ordering and balance.

6.2 Whole-body reaching: the important difference is moving the body

The reaching benchmark contains 54 wrist targets spanning left and right arms, three distances from 0.35 to 1.8 meters, and three heights from 0.55 to 1.35 meters. A trial succeeds when the wrist enters a 10-centimeter region and remains there while the robot stays upright. For methods that succeed, hand precision is similar. The main separation is whether the system can step, lean, and counterbalance enough to make the target reachable.

Method	Reach success	Error	Upright
METATRON	83%	5.9 cm	100%
ARDY + SONIC	39%	6.4 cm	82%
GRU + IK + SONIC	28%	5.6 cm	81%
GRU + SONIC	0%	-	87%

The result rules out a purely kinematic interpretation. A high target may require the torso to move backward as a counterweight. A low target may require hip flexion and a different support configuration. A reference generator that keeps the standing height fixed can ask the arm to reach a point that is physically impossible without changing the base. ARDY can satisfy hand goals through classifier guidance, but the paper reports that the wrist can oscillate across repeated replanning. METATRON receives the hand target as part of the policy condition and resolves it together with stepping and torso motion.

The upright column also matters. A controller that reaches a target by falling over has not solved the task. METATRON reports 100 percent upright retention, compared with 82 and 81 percent for the two partially successful baselines. The result is still simulation evidence, and the target distribution is finite. It should not be generalized to arbitrary reachable space, but it is consistent with the architectural claim that whole-body coordination must be optimized jointly rather than delegated to arm inverse kinematics after a fixed base.

6.3 Push recovery reveals a large gap in the tested pipelines

The robot walks forward at one meter per second while a 100-millisecond force impulse is applied from one of eight directions. A trial counts as successful if the robot is still upright after eight seconds. The comparison uses two reference generators followed by the same SONIC dense tracker. This METATRON variant intentionally omits synthetic recovery training so the experiment isolates the control interface from the full training recipe.

Method	200 N	400 N	600 N	800 N	1000 N
METATRON	100%	100%	62%	75%	75%
ARDY + SONIC	100%	100%	17%	0%	0%
GRU + SONIC	100%	75%	25%	0%	0%

The non-monotonic sequence between 600 and 1000 newtons should not be read as a claim that larger forces are easier. The defensible conclusion is that METATRON remains successful in 75 percent of trials at both 800 and 1000 newtons, while both reference-tracking pipelines have reached zero above 600 newtons. Because the comparison changes an entire control pipeline, it does not isolate whether the prompt interface, distillation objective, or data distribution produces the difference. The experiment is evidence for the system-level design, not a controlled causal test of one component.

Figure 4: Real G1 recovery frames from paper Figure 7. Synthetic recovery data expands the range of ground configurations that are rare in motion capture.

6.4 Ablation: recovery data and auxiliary losses help at moderate perturbations

The ablation uses planar waypoint targets without the gait-generated foot targets or height constraints. A horizontal push is applied every one to two seconds during the first five seconds. Success requires entering a 0.5-meter goal region within ten seconds, keeping normalized torso height above 0.45, and limiting tilt to 60 degrees. Each force and method combination is evaluated over 500 episodes.

Method	0 N	200 N	400 N	600 N	800 N
METATRON	97.4%	94.8%	33.0%	7.4%	6.2%
Global only	87.6%	87.8%	38.2%	9.6%	6.2%
Mocap only	92.0%	84.8%	31.8%	6.4%	5.0%
No aux.	81.6%	78.8%	29.2%	5.4%	4.6%
MaskedMimic+	98.2%	90.0%	27.8%	4.2%	4.6%
HOVER+	66.8%	71.0%	30.0%	3.0%	1.2%

The full model is slightly below MaskedMimic+ at zero force but exceeds it at every nonzero perturbation level in the table. Its largest gains over the no-recovery and no-auxiliary variants occur between 0 and 200 newtons. Global only is higher at 400 and 600 newtons, although the confidence intervals overlap, so the authors do not claim a decisive advantage there. Every method falls below 10 percent at 600 newtons. Strong pushes remain outside the demonstrated reliability envelope.

The ablation is useful because it separates "the robot can follow a waypoint when undisturbed" from "the robot can recover while pursuing that waypoint." The zero-force difference is small relative to the changes under perturbation. Recovery data and auxiliary losses improve the behavior that appears when the body is already departing from the nominal gait. The result does not prove that those terms solve arbitrary contact, but it identifies where they matter in this distribution.

6.5 SOMA demonstrates composition, not statistical generalization

A SOMA character is trained with a similar recipe and runs, slides down a step, recovers after falling, and climbs a wall in simulation. The skill changes velocity, heading, pelvis height, and temporary foot targets across the phases. During the climb, it provides a pelvis position and orientation above the ledge; the policy discovers the hand and foot contacts. This demonstration supports a claim of transfer across embodiments and more dynamic scenarios, but it is not accompanied by the same multi-condition success statistics as the G1 reaching and push experiments.

Figure 5: The SOMA motion sequence from paper Figure 6. A state machine changes velocity, pelvis, and climbing targets while the frozen policy supplies contacts and recovery.

Figure 6: A physical furniture interaction from paper Figure 5. An external system supplies coarse sofa geometry, and the skill generates torso and foot targets.

7. How the public implementation maps to the paper

The official code lives under ProtoMotions v3 and includes G1 and SOMA training entry points, pretrained G1 weights, MuJoCo inference configurations, TensorRT export, and a native C++ deployment reference. The source is Apache-2.0, while the pretrained model is released under the NVIDIA Open Model License. The paper's G1 student is distilled from four expert groups. The public base-policy examples emphasize a single-teacher recipe, while the pretrained G1 scene-interaction policy is distributed separately in the model package. Readers should distinguish an available checkpoint from a complete reproduction of the training corpus and multi-teacher pipeline.

7.1 Five future slots and multiple prompt streams

The G1 student configuration mirrors the sparse interface described in the paper. It sets the number of masked future steps to five and creates separate observation components for world targets, local targets, velocity, style, and scene information:

control = MaskedMimicControlConfig(
    num_masked_future_steps=NUM_FUTURE_STEPS,
    sparse_target_condition_probability=0.5,
    local_target_condition_probability=0.5,
    anchor_pose_submasking=True,
    style=MaskedMimicStyleConfig(
        visible_frame_count_probabilities=(0.50, 0.15, 0.15, 0.10, 0.07, 0.03)
    ),
    velocity_command=MaskedMimicVelocityCommandConfig(horizon_seconds=0.5),
)

Source: examples/experiments/metatron/g1_student.py:193-204. The same configuration distinguishes current state, history, global tokens, local tokens, style tokens, and velocity tokens. A six-layer, four-head, 1024-dimensional Transformer fuses them. The paper's use of the word prompt should therefore not be confused with natural-language prompting. The runtime prompt is a collection of numeric tensors and explicit masks, even though the interface may later be wrapped by a language model or task planner.

7.2 Spherical latent and deployment-only prior

The configuration explicitly selects a 64-dimensional spherical latent, a unit-mean penalty, and a cyclical KL schedule:

vae=MaskedMimicVAEConfig(
    vae_latent_dim=64,
    latent_geometry=LatentGeometry.SPHERE,
    prior_mean_unit_norm_coeff=0.001,
    kld_schedule=KLDScheduleConfig(
        start_epoch=500, end_epoch=2000, cyclical=True
    ),
)

Source: examples/experiments/metatron/g1_student.py:511-527. The model forward path runs the prior and decoder, while the training path can also run the privileged encoder. The deployment entry point forward_deployable calls only the prior and decoder. The export wrapper in deployment/metatron_policy.py:15-35 goes further and fixes the noise to zero, retaining a deterministic prior path. This is consistent with the physical demonstrations, which use deterministic inference rather than sampling several possible motions.

The native recovery implementation is more elaborate than the high-level description in the main paper. deployment/native/src/metatron_v1_recovery.cpp:170-223 separates prone, seated, ramp, and settle phases and uses orientation, knee angles, and hand displacement to decide when to use a reference recovery clip. The paper describes the conceptual state transitions; production code adds thresholds, deliberate low-posture checks, and stabilization time. A reproduction based only on the paper's single height threshold would not match the released behavior.

8. Limitations and deployment risks

The authors state that the motion library determines the capability boundary. A sparse prompt can be ambiguous, and style frames can occasionally introduce an unintended bias. A sudden command from standstill to two meters per second often causes a forward roll. The deployment code responds by raising the velocity target through exponential smoothing. This example shows that freezing the policy does not eliminate tuning. The skill must still understand the policy's preferred action distribution and safety envelope.

The task system remains responsible for perception and long-range planning. In the sofa experiment, the robot is placed near the furniture and receives a seat box directly. The system does not discover the sofa, navigate through a cluttered room, estimate every obstacle, or plan a multi-step household task. Recovery is evaluated from a finite set of initial configurations. The paper does not provide large-sample reliability statistics across floor materials, actuator wear, payload changes, or repeated falls.

Physical evidence is mainly qualitative. Walking quality, compliant handshaking, and furniture interaction are shown on hardware, but they do not receive the same multi-condition confidence intervals as reaching and push recovery. The quantitative experiments are primarily in MuJoCo. The 83 percent reaching success therefore should not be treated as the probability of reaching arbitrary objects in an open room. It measures performance on a defined target grid with a simulated evaluation protocol.

Three independent risks also deserve attention. First, world-frame targets require localization, and absolute height targets require a ground reference. Localization drift can become a stable action bias. Second, scene prompts depend on premeasured boxes and pose estimates. Deformable objects, transparent surfaces, and crowded human environments remain difficult. Third, coding-agent iteration requires a reliable success metric and a representative simulation suite. An agent can optimize a program that passes the available tests while remaining unsafe in an unmodeled situation.

Training cost is substantial. The student uses 20 GPUs in the paper's recipe, and the teachers add further training. The published ablation uses eight A100 GPUs for 5000 iterations. For many robotics teams, fine-tuning or writing skills around the released G1 checkpoint is more realistic than training a multi-expert humanoid foundation policy from scratch. The paper's architecture is reproducible in principle, but the full data and compute pipeline is not a small local experiment.

9. Discussion: the durable contribution is an evolvable interface

METATRON is more than another motion tracker because it exposes learned motor competence through a stable set of composable and inspectable interfaces. A task engineer can inspect whether future targets are reasonable, whether masks cover the intended bodies, and whether a state machine changes modes under the right conditions. These checks are ordinary software tests. The neural policy remains difficult to interpret, but its role is constrained to completing a sketch rather than deciding the full task.

This separation changes the economics of data collection. A new motor capability may require more expert data and another distillation round. A new task within the existing capability set should first be attempted as a program. Only when the required action is absent should the team return to data collection and policy training. Teams can then distinguish a coverage failure from a task-logic failure. That diagnostic clarity is valuable even when the particular G1 checkpoint is not used.

The next step is a tighter perception loop. Current targets come from controllers, headsets, waypoints, or precomputed boxes. If a vision-language model or geometric perception system can continuously provide object boxes, free-space regions, and human intent, METATRON could become part of a more autonomous humanoid stack. The skill layer would also need reachability checks, target smoothing, safety constraints, and explicit fallback states. A success-rate benchmark alone is insufficient for physical deployment.

Another direction is to let coding agents construct larger state graphs that sequence running, climbing, grasping, and recovery. The paper demonstrates that program iteration can improve gait and sitting behavior, but it does not show that an agent can discover goals, define its own safety tests, or infer general motor laws from failures. The upper bound of the skill layer still depends on the action representation, simulator fidelity, and evaluation design.

10. Conclusion

METATRON proposes a clear division of labor for humanoid control. Dense experts learn high-quality whole-body coordination. A generative student compresses their behavior into a policy that accepts sparse prompts. The resulting controller is frozen, while ordinary programs express task intent and switch modes. Spherical latents, privileged posterior distillation, synthetic recovery, scene-aware retargeting, and multi-stream masking are the mechanisms that make this division work.

The experiments show that one frozen G1 checkpoint can cover several control modes and physical tasks, and that it outperforms the tested reference-tracking pipelines on whole-body reaching and push recovery. The same evidence also exposes the boundary: motion coverage, perception, long-horizon planning, hardware statistics, and strong perturbations remain open. METATRON is best understood as a credible execution-layer architecture and a useful engineering base, not as a complete solution to general humanoid autonomy.

Key insight: Let the base policy learn how to coordinate a body, and let programs state what the body should achieve. A new task then becomes a new skill, not necessarily a new controller.