MetaTron: A Promptable Whole-Body Execution Layer for Humanoids

MetaTron's central bet is architectural, not merely algorithmic: stop retraining a humanoid controller for every task. Train one whole-body execution layer, freeze it, and let ordinary programs specify what the body should accomplish next.

Humanoid robots already have impressive demonstrations of walking, reaching, standing up, dancing, and recovering from a fall. The difficulty is that these abilities are often delivered by separate controllers, each with its own reward, observations, checkpoints, and transition logic. Every new behavior then asks the same underlying questions again: how should the body shift its weight, coordinate the limbs, and stay stable?

MetaTron, a research project from NVIDIA with Dollhouse Robotics, proposes a cleaner division of labor. A generative policy handles physical coordination. Stateful code called a skill supplies intent. The policy is trained once for each embodiment and then frozen. New behaviors are added by changing the goals sent to that policy, not by training another motor controller.

The project's shorthand is deliberately blunt: "Robot control is now an LLM tool call." That slogan needs one important qualification. The language model does not run inside the 50 Hz balance loop. It writes and iterates on skill code offline. At runtime, the skill is called like any other software tool, and that code sends structured goals to MetaTron.

The problem: every task wants its own controller

A conventional humanoid stack may use one policy for locomotion, another for manipulation, another for getting up, and yet another for compliant physical interaction. These systems duplicate balance and whole-body coordination. They also create seams at exactly the moments that matter: sitting down and then standing, recovering and then walking away, or holding a torso goal while a person moves the robot's arms.

MetaTron instead separates two questions:

• What should happen? A hand should reach a point, the torso should lower onto a seat, the robot should move at a certain velocity, or the body should become upright.
• How should the body move? The learned policy decides how to distribute the motion across joints, maintain balance, and produce actions.

The first question belongs to a skill. The second belongs to the execution layer. A skill can be simple code, a finite-state machine, or a periodic trajectory rule. It does not need to plan every joint. MetaTron learns to tolerate approximate or incomplete body goals, so the skill can sketch a desired motion rather than reproduce a perfect reference trajectory.

This is not a claim that planning disappears. The task system still chooses goals and sequences long-horizon behavior. Perception, localization, object detection, and autonomous approach remain separate problems. The contribution is narrower and more useful: once a behavior can be expressed as a sequence of partial body goals, the same motor policy can carry it out.

A prompt is not text: it is a structured motion goal

MetaTron uses the word "prompt" in a control-specific sense. At time $t$, the prompt is a collection of optional goal streams and scene context:

$$g_t = \left(C_t^w,\ C_t^\ell,\ u_t,\ \xi_t,\ b_t\right)$$

Each term answers a different kind of request:

• $C_t^w$ contains body positions or orientations in world coordinates, each with a time of arrival.
• $C_t^\ell$ contains body poses in a canonical local frame that is independent of world position and heading.
• $u_t$ is planar travel velocity in the robot's heading frame.
• $\xi_t$ is a style cue made from a small, unordered set of example poses.
• $b_t$ describes nearby scene geometry as boxes, represented by their eight corners. A sofa, for example, can become a seat box and a backrest box.

Masks determine which bodies, properties, and future times are constrained. A goal can request only hand position, leaving hand orientation free. It can specify a torso position but leave all limb targets unspecified. It can combine world-space objectives with local style cues. Even time can be approximate because each target carries an arrival time rather than an exact frame index.

This interface is more expressive than velocity alone and more forgiving than dense motion tracking. A velocity-only command chooses how far and how fast, but says nothing about foot rhythm. A dense reference specifies every body over time, but leaves little room for adaptation. MetaTron's prompts sit between them: enough structure to communicate intent, enough freedom for the learned policy to coordinate the solution.

Stage one: train dense motion specialists

Training begins with reinforcement learning in simulation. A motion tracker receives privileged simulator state and a complete future reference. It outputs joint actions that reproduce that reference. For the Unitree G1, NVIDIA trains specialists on four data sources:

• BONES-SEED for large-scale everyday motion capture.
• AMASS + LAFAN for locomotor, athletic, and general human movement.
• Synthetic recovery motions generated from randomized fallen configurations.
• SAMP scene interactions, retargeted with collision-aware optimization for sitting on furniture and similar behaviors.

The four G1 expert groups are assigned relative training weights of 0.6, 0.2, 0.1, and 0.1. Each group has a 32-step rollout, 2,048 environments per GPU, and two optimization passes. The experts are trained with PPO. Their reward combines exponential tracking terms for body position, orientation, and velocity, with penalties for abrupt action changes and embodiment-specific failures:

$$r_t = \sum_k w_k\exp\left(-\lambda_k e_{k,t}\right)-w_a R_{\mathrm{act},t}+r_t^{\mathrm{extra}}$$

The extra reward includes selected-body precision terms, joint-limit penalties, contact mismatch, and self-contact penalties. These rewards train the dense trackers, not the deployed student. They are the scaffolding that makes the second stage possible.

Stage two: distill complete motion into partial-goal control

A partial prompt admits many valid completions. Regressing one action directly from the prompt can average incompatible motions into something that looks indecisive or unstable. MetaTron therefore uses a conditional variational autoencoder. A training-only posterior sees the full reference and privileged state. A deployable prior sees only observations, sensor history, and the partial prompt.

For observations $x_t=(o_t,a_{t-1})$, privilege state $s_t$, and full reference $y_t$, the two distributions are:

$$p_\theta(\tilde z_t\mid x_t,g_t)=\mathcal N\left(\mu_t,\operatorname{diag}(\sigma_t^2)\right)$$

$$q_\phi(\tilde z_t\mid x_t,g_t,s_t,y_t)=\mathcal N\left(\mu_t+\Delta\mu_t,\operatorname{diag}(\sigma_t'^2)\right)$$

The sampled latent is projected onto the unit sphere, then fused with a deterministic condition:

$$z_t=\frac{\tilde z_t}{\lVert\tilde z_t\rVert_2},\qquad h_t=f_\theta(d_t,z_t)$$

The condition $d_t$ carries the observed context and prompt. The sphere-projected latent selects a compatible motion completion. The decoder combines both with the current observation and previous action to produce joint targets. At deployment, the posterior, the full-reference encoder, and the dense expert are removed. Only the prior, fusion module, and action decoder remain.

Distillation is on-policy. The student drives most simulation rollouts, and the frozen tracker labels the states that the student actually visits. This matters because a student makes mistakes. A dataset built only from perfect reference states would not teach it how to recover from its own deviations. The training mixture assigns control to the posterior-guided student with probability 0.9 and to the frozen tracker otherwise.

The main objective combines action imitation, prior-posterior alignment, unit-norm regularization, dynamics prediction, temporal consistency, and action smoothing:

$$\mathcal L=\mathbb E\left[\ell(a_t^q,a_t^{\mathrm{trk}})\right]+\beta_eD_{\mathrm{KL}}(q_\phi\Vert p_\theta)+\lambda_\mu\mathbb E\left[(\lVert\mu_t\rVert_2-1)^2\right]+\mathcal L_{\mathrm{aux}}$$

The auxiliary terms predict the next condition, smooth consecutive fused representations, and smooth consecutive actions. The G1 model uses cyclical annealing for the KL weight. At inference, only the prior path runs.

One policy is produced for the G1 and a separate policy for the simulated SOMA character. The G1 policy distills four experts into one student; SOMA uses a scene-aware tracker. This is why the work can demonstrate both a physical humanoid and simulated running, sliding, climbing, and acrobatics without claiming that one checkpoint transfers across embodiments.

Stage three: skills are ordinary code around a frozen policy

After distillation, MetaTron stops learning. Every new behavior is a skill: code that observes task state and updates the prompt at each control step. A skill can use a clock, a controller input, a finite-state machine, object geometry, or a recorded motion. The policy remains unchanged.

The key design choice is that skills do not need to command joints. They add goals that make an underspecified motion easier to execute. A velocity command alone may produce short, hesitant steps. A steering skill adds timed foot-placement targets and yields a more committed gait. The policy still handles balance and joint coordination.

For a foot $i$ at future time $\tau$, the paper's steering rule places a target at:

$$p_i(\tau)=b_i+\begin{bmatrix}A(v)\cos\alpha\cos\phi_i\\W s_i|\sin\alpha|+|\cos\alpha|\sin\alpha\cos\phi_i\\H\max(0,-\sin\phi_i)\end{bmatrix}$$

Here $b_i$ is the calibrated stance offset, $v$ is travel speed, $\alpha$ is the angle between travel and facing, and $\phi_i$ advances with a speed-dependent cadence. The coefficients make steps longer and quicker as speed rises, widen the stance during lateral travel, and lift the foot at mid-swing. The two feet are half a cycle apart.

A simplified version of the deployment logic looks like this:

frequency = min(2.0, 0.83 + 0.93 * speed)
amplitude = 0.13 + 0.065 * speed
phase += 2 * pi * frequency * dt

for future_time in future_times:
    for foot, phase_shift in [(right_foot, 0), (left_foot, pi)]:
        angle = phase + 2 * pi * frequency * future_time + phase_shift
        offset = stance_offset(foot, travel_angle)
        offset.xy += amplitude * cos(angle) * swing_axis
        offset.z += 0.08 * max(0, -sin(angle))
        goals.add(foot, to_world(offset, future_time), future_time)

policy.execute(velocity, heading, goals)

This is still a program. It does not learn a task-specific policy. It does not bypass the controller and write joint commands. It changes the prompt so the same policy chooses a better walking solution.

Sitting: three coarse targets are enough

The sitting skill receives a nearby-object cue encoded as boxes. It extracts a torso target above the seat and foot targets just beyond the front edge. The torso target is static for the downward motion. To stand, the skill raises the torso along a straight line and moves it forward, while the feet and balance policy fill in the stepping pattern.

On hardware, the robot starts near the chair or sofa and receives the object representation. Autonomous perception and approach are not demonstrated. The relevant result is that the same frozen controller can switch from standing and walking to sitting, then stand and walk away, with only the prompt changing.

Recovery: change the goal when the body falls

Recovery is a state-machine skill. It compares current torso height with the upright reference. Below 65% of upright height, the skill suspends the task prompt and requests an upright pose. Above 90%, it restores the task. The hysteresis prevents rapid switching near the boundary.

The first requested arrival time scales with the remaining height gap:

$$\tau_0(h)=\operatorname{clip}\left(\frac{H-h}{v_{\mathrm{rise}}},0.25,1.0\right)$$

For simulation, $v_{\mathrm{rise}}=0.8$ m/s. Later target slots add 0.25 seconds, up to a two-second horizon. The G1 deployment skill simplifies the request further: it specifies torso height and leaves position, orientation, hands, and feet largely free. The hardware demonstrations show recovery from seated, back, side, and stomach configurations. They are evidence of a repertoire, not a safety guarantee.

Compliance: leave the arms unconstrained

For physical guidance, the skill commands zero travel velocity, a torso height, and a heading, but says nothing about the arms. Because the policy treats unspecified properties as free, a person can move the robot's arms without fighting a hidden full-pose target. The project demonstrates handshakes and guided motion, while explicitly noting that this is not a certified safety guarantee.

Agent-written skills: code is the new task interface

The paper describes a skill-authoring loop in which a human states the behavior and success criteria, and a coding agent builds a simulation playground, analyzes motion clips, proposes compact rules, tests them, and revises failures. The agent changes goal-generating code, not policy weights.

NVIDIA's project page gives two concrete examples. Claude Code was used to fit the steering rule from motion data. For the simulated sports tasks, GPT-6 Astra, with a human directing, wrote and tuned the goal code, timing, and motion references.

This workflow is interesting because it changes what "tool use" means for a robot. A conventional tool call might be `open_cabinet()`, but MetaTron's skills are lower in the stack: they produce timed body targets, velocity commands, and transition logic. The language model is useful because it can write and iterate on that code against simulation results. It is not asked to infer torques from pixels at runtime.

The sports demonstrations show the flexibility of this separation:

• Four hurdles: the robot runs, switches to recorded jump goals 2.6 meters before each bar, and resumes running once both feet clear it. The shown run clears four 20 cm hurdles with zero bar contacts.
• Fall and recover: a 40 cm bar causes a fall at 3.83 m/s. The same skill detects the low torso, switches to an upright request, and returns to running 3.28 seconds after the fall.
• Bowling: a recorded motion supplies body targets and a timed ball release. The shown run knocks down 9 of 10 pins.
• Tennis and football: timed racket or foot targets meet a scripted ball. The tennis ball leaves at 5.37 m/s; the football reaches 4.82 m/s. These are timed drills, not visual perception.

The hardware stack

The deployed G1 policy is implemented in C++. TensorRT and CUDA graphs run the network on a laptop with an NVIDIA RTX 5090 GPU, connected to the robot by Ethernet. An Xbox controller supplies steering commands, while a PICO headset maps hand motion for teleoperation.

Sensor-to-command latency is measured at 1.3 ms. That number is processing latency, not the control period. The policy produces a new action every 20 ms, or 50 Hz, with four simulator physics steps per action during training. Hardware demonstrations use the deterministic prior mean rather than sampling a latent.

One checkpoint supports steering, teleoperation, sitting, recovery, compliant guidance, and dense motion playback. Switching modes changes the goal observations and masks. It does not swap model weights or sensor-processing code. That is the practical meaning of an "execution layer": a stable motor substrate on which task-specific programs can be composed.

What the experiments actually show

MetaTron is compared with two reference-tracking pipelines on the same G1. One generates whole-body references with ARDY and tracks them with SONIC. The other uses a GRU locomotion generator, adds analytic arm IK for reaching, and tracks the result with SONIC. These comparisons test complete pipelines, so they do not isolate a single cause for every difference.

Whole-body reaching

The reaching test uses 54 wrist targets at three distances from 0.35 to 1.8 meters, three heights from 0.55 to 1.35 meters, and both hands. A target counts only if the wrist remains within 10 cm while the robot stays upright.

Method	Targets reached	Error on reached targets	Upright
MetaTron	87%	5.7 cm	100%
ARDY + SONIC	39%	6.4 cm	82%
GRU + IK + SONIC	28%	5.6 cm	81%
GRU + SONIC	0%	Not available	87%

The interesting split is not arm precision. Among methods that reach a target, wrist error is similar. The difference is whether the whole body can adapt enough to get there at all. High targets require counterbalance; low targets require a hip fold. A generator that plans at a fixed standing height cannot easily provide either.

Push robustness during steering

Each trial applies a 100 ms force pulse to the torso while the G1 walks forward at 1 m/s. Forces span 200 to 1,000 N across eight directions. Success means remaining upright through the final two seconds of an eight-second window.

Force	MetaTron	ARDY + SONIC	GRU + SONIC
200 N	100%	100%	100%
400 N	100%	100%	75%
600 N	38%	17%	25%
800 N	50%	0%	0%
1,000 N	88%	0%	0%

The reference pipelines fail all trials above 600 N. MetaTron survives seven of eight directions at 1,000 N. The non-monotonic curve should not be read as a physical force-response law: the paper uses one trial per direction, and the 600, 800, and 1,000 N differences are within noise for this sample size.

Dense motion tracking

A promptable policy should still be able to follow a complete reference when that reference is available. The benchmark contains 129,553 BONES-SEED clips totaling 262 hours. Both MetaTron and SONIC receive the reference 0.9 seconds ahead in MuJoCo.

Protocol and method	Success	Position error	Rotation error	Jerk
Global, MetaTron	88.5%	42 mm	10.7 deg	123 m/s3
Global, SONIC	49.6%	83 mm	9.9 deg	240 m/s3
Local, MetaTron	97.9%	35 mm	11.4 deg	111 m/s3
Local, SONIC	98.7%	33 mm	10.6 deg	146 m/s3

Under SONIC's local protocol, the two policies are effectively matched on success and pose error, while MetaTron reports 24% lower jerk. Under the global protocol, MetaTron is considerably better at staying where the reference is in the world. That comparison also tests the interface: SONIC receives no world-position goal and cannot explicitly correct accumulated drift. The result matters for a controller that is supposed to accept "go there" prompts, but it is not a pure tracker-versus-tracker result.

MetaTron is also not tied to a 0.9-second lookahead. With a 90 ms horizon, the same checkpoint still succeeds on 94.8% of clips under the local protocol and 54.0% globally. The latter is already above SONIC's global success at the full 0.9-second horizon.

What MetaTron does not solve

The limitations are as important as the demonstrations:

• The repertoire follows the training data. If the motion library does not cover a behavior, the frozen policy cannot invent a reliable solution merely because a skill asks for it.
• Underspecified goals can select the wrong motion. Abruptly requesting 2 m/s from rest often produces a front roll. NVIDIA smooths velocity commands with an exponential moving average, but this is mitigation, not a general solution.
• Task selection remains external. MetaTron does not plan a long-horizon task or decide which skill to call. The surrounding agent or state machine does.
• Perception is not part of the demonstrated sitting loop. The robot is placed near the furniture and receives a supplied object cue. Autonomous detection, localization, and approach remain future work.
• The sports examples use task code, not visual perception. Hurdle locations and ball timing are scripted in simulation. The results demonstrate whole-body execution and skill composition, not autonomous sports playing.
• Hardware evidence is demonstrative. Recovery is shown from tested configurations, and the gait improvement is assessed qualitatively. Reliability, contact safety, and behavior near actuator limits still require systematic measurement.

These caveats do not erase the architectural result. They define where it sits: MetaTron is a programmable motor layer, not an autonomous robot brain.

Why this matters for embodied AI

Much of the current robotics stack tries to make one large model reason about language, vision, planning, and low-level control at once. MetaTron argues for a different boundary. The language model can be useful without being the controller. It can inspect motion data, write goal-generating code, build simulation tests, and debug failures at a timescale of seconds or minutes. A smaller frozen policy handles balance and coordination at 50 Hz.

That separation has three practical advantages. First, a new task does not require a new motor policy. Second, the task logic is readable and versionable as code. Third, the expensive learned component can be trained and validated once per embodiment, while application-specific behavior evolves faster around it.

The approach also exposes the next bottleneck clearly. If skills are the new interface, then robots need reliable ways to generate, test, and certify those skills. A coding agent can iterate in simulation, but simulation coverage, perception, safety, and real-world variation remain hard. The skill layer makes those problems explicit instead of burying them inside another end-to-end network.

MetaTron is therefore best understood as infrastructure. It does not make every humanoid behavior a language-model output. It turns many behaviors into code that a language model can help write, provided the desired motion lies within the controller's learned repertoire and the surrounding system can supply the right goals.

Sources and further material

• Project page: NVIDIA Research - MetaTron
• Technical paper: MetaTron: A Promptable Whole-Body Execution Layer for Humanoids
• Interactive MuJoCo demo: ProtoMotions MetaTron demo

Source: NVIDIA Research, MetaTron. Adapted for RobotWorld from the project page, interactive explanations, and technical paper. Media is hosted locally from the project's published assets.