PAPER DEEP DIVE
UniWAM: Unified World-Action Model
Vision-language-action models benefit from the understanding and reasoning capabilities of pretrained vision-language models, but action-only supervision provides limited grounding in world dynamics. Conversely, world-action models inherit spatiotemporal priors from video generation models, yet remain limited in semantic understanding and reasoning under distribution shifts. We introduce UniWAM, a unified architecture that integrates a physical reasoner, a world generator, and an action predictor to jointly learn semantic understanding of the physical world, visual generation, and action prediction. To ensure the quality of the training data, we developed a rigorous data cleaning and annotation pipeline for both human egocentric data and robot data. To adapt the vision-language component to embodied tasks while preserving its inherited language capabilities, we represent low-level actions in natural language and introduce a pre-training recipe that assigns complementary supervision from visual question answering (VQA) data, human egocentric data, and robot demonstrations to the appropriate model components. During post-training, future visual noise augmentation reduces reliance on precise future predictions, while history-conditioned flow matching uses encoded action history to initialize action generation. Together, these designs significantly reduce denoising steps while maintaining performance. UniWAM achieves state-of-the-art (SOTA) performance across multiple evaluations, including in-distribution performance, robustness, generalization, instruction following, and long-horizon task execution. Furthermore, we uncover a log-linear scaling law of unified human-robot co-training, demonstrating the effectiveness of large-scale pre-training on a mixture of human and robot data.
One-Sentence Summary
UniWAM couples a vision-language reasoner, a video world generator, and a continuous action expert through joint attention at every decoder layer, then uses physical-language targets and human-robot co-training to make semantic reasoning and future dynamics useful for manipulation.
Research Context: Two Kinds of Competence That Rarely Meet
Vision-language-action models inherit broad object knowledge, scene understanding, and instruction following from image-text pretraining. Their control training, however, is usually dominated by paired observations and robot actions. The action label says what the robot did, but it does not explicitly explain how the scene will change after contact. A model can therefore fit the training trajectories while remaining vulnerable to visual shortcuts, especially when backgrounds, lighting, object layouts, or robot initial states differ from the demonstrations.
World-action models take a different route. A video-generation backbone has seen extensive spatiotemporal data and can represent motion, object persistence, and plausible consequences. This is valuable prior knowledge for control because an action is not merely a point in a regression space; it changes the future state of the world. Yet a generative video model does not automatically know which object a compositional instruction refers to, which relation is semantically relevant, or how a verb should change the interaction pattern.
The paper describes these weaknesses as complementary. Vision-language policies have semantic bandwidth but weak explicit dynamics supervision. World-action models have dynamic bandwidth but weaker language grounding and compositional reasoning. UniWAM attempts to combine them without reducing one system to a feature extractor for the other. All three experts remain active in a shared transformer stack, and their tokens exchange information repeatedly rather than only at the input or final head.
This architectural choice matters because late fusion postpones the problem. If a video model first produces a future clip and an action model then consumes that clip, the action expert cannot influence what the video model attends to while its latent representation is being formed. Conversely, if a vision-language model emits one fixed semantic vector, the video and action experts cannot ask for finer object or contact details in later layers. UniWAM instead places understanding, video, and action tokens in one joint attention operation at each layer.
The training problem is equally important. Visual question answering data can preserve semantic competence but contains no robot action labels. Human egocentric video contains rich manipulation and world dynamics but lacks reliable robot-specific low-level actions. Robot demonstrations provide precise state-action supervision but are much smaller and less diverse than web-scale video. The paper does not force every data source to supervise every output. It assigns complementary targets to the three experts, which is why the data recipe is as central as the network topology.
A second training problem is inference efficiency. Flow matching can require multiple integration steps. If every step also runs language, video, and action tokens through a large joint stack, deployment becomes expensive. UniWAM addresses this in post-training by perturbing future visual latents and by initializing action flow from a slightly noisy action-history chunk. The first reduces dependence on an unrealistically clean future; the second turns temporal continuity into a useful starting point instead of rebuilding a motion chunk from Gaussian noise.
Figure 1: The paper's overview. Diverse robot, human, and vision-language sources are connected to a three-expert model whose outputs cover physical language, future video, and actions. Production asset derived from paper Figure 1.
Formalizing Understanding, Imagination, and Control
At control time, the model receives an observation $o_t$, a proprioceptive state $s_t$, and an instruction $I$. It must produce a language sequence $y$, a short future observation sequence, and an action chunk. The paper writes the target as a single conditional distribution:
$$ p_{\theta}\left(y,o_{t+1:t+h},a_{t+1:t+h}\mid o_t,s_t,I\right). $$
Here $y$ may be an answer in visual question answering or a physical-language description of local motion, $h$ is the horizon, and $\theta$ contains the trainable parameters. The equation is not a claim that all outputs are decoded by one homogeneous head. It states that the three predictive problems share a conditioning context and a learned representation. The implementation realizes this sharing through modality-specific experts and masked joint attention.
The physical reasoner is initialized from Qwen3-VL-2B-Instruct and uses a hidden width of 512. The world generator is initialized from Wan2.2-TI2V-5B, with hidden width 3,072 and 24 attention heads. The action predictor uses hidden width 1,024. Each expert has 30 layers. The control interface is 14-dimensional, the model predicts eight future frames at $384\times320$, and the action chunk length is 16.
For modality $m\in\{u,v,a\}$, where $u$, $v$, and $a$ denote understanding, video, and action, respectively, the experts first normalize their tokens. Video and action tokens also receive timestep modulation. They are then projected into a common attention space:
$$ Q_m=\widetilde X_m^{\ell}W_m^Q,\qquad K_m=\widetilde X_m^{\ell}W_m^K,\qquad V_m=\widetilde X_m^{\ell}W_m^V. $$
The projections are concatenated along the token axis, and all modalities participate in one masked scaled-dot-product attention operation:
$$ O=\operatorname{softmax}\left(\frac{QK^\top}{\sqrt d}+M\right)V, \qquad Q=[Q_v;Q_a;Q_u]. $$
$K$ and $V$ use the same modality order, $d$ is the dimension of one head, and $M$ masks invalid attention connections. The output is split back into video, action, and understanding streams before expert-specific output projections, residual paths, and feed-forward blocks. Because this happens in every one of the 30 layers, the action stream can expose control intent to the visual stream early enough to affect the evolving future representation.
This differs from an ensemble of independently trained models. The experts retain separate weights, widths, and normalization behavior, but their exchange is not a one-time adapter. The shared attention space is also where the paper's claim becomes testable: if the world generator is irrelevant, zeroing or corrupting future visual tokens should have little effect; if language grounding is irrelevant, training the reasoner should add less than the ablations report. The experimental section provides evidence on both points.
flowchart TB
O["Observation o_t"] --> PR["Physical Reasoner
Qwen3-VL 2B
width 512"]
I["Instruction I"] --> PR
O --> WG["World Generator
Wan2.2 TI2V 5B
width 3072 / 24 heads"]
S["State s_t"] --> AP["Action Predictor
width 1024"]
H["Action history chunk"] --> AP
PR --> J1["Joint attention layer 1"]
WG --> J1
AP --> J1
J1 --> J2["Layers 2 through 30"]
J2 --> L["Physical language / VQA head"]
J2 --> V["Video flow head
8 future frames"]
J2 --> A["Action flow head
14 dimensions / chunk 16"]
V -. "future latent feedback" .-> J2
A -. "control-intent feedback" .-> J2
The diagram reflects the paper's specific architecture: three 30-layer experts, hidden widths 512, 3,072, and 1,024, 24 world-generator heads, eight future frames, and a 16-step action chunk. It is not a generic multimodal pipeline.
Physical Language as the Bridge Between Embodiments
Directly regressing a 14-dimensional continuous action can move a pretrained language model away from its native token distribution. UniWAM instead converts local end-effector motion into structured language following the language-action pretraining idea. Translation is rounded to centimeters, rotation to ten-degree intervals, and signs along the three axes map to forward, backward, up, down, left, and right. Orientation changes map to roll, pitch, and yaw descriptions. Unchanged dimensions are omitted, while the gripper is described as open or closed using a 0.5 threshold. For dual-arm data, the descriptions receive left-arm or right-arm prefixes.
The template also creates a shared target for humans and robots. Human egocentric data may lack precise robot joint commands, but wrist position, orientation, and gripper state can still produce the same kind of local motion sentence. Task instructions describe the overall goal, while physical language describes the behavior associated with the current observation. The reasoner therefore learns two levels of language: what the task asks for and what physical event should happen next.
The language objective is the usual autoregressive negative log-likelihood over answer tokens:
$$ \mathcal L_{\mathrm{lang}} =-\sum_{j=1}^{T}\log p_\theta \left(y_j\mid o_t,I,y_{<j}\right). $$
The loss is applied to the answer portion, not to reconstructing the prompt. For VQA, $y$ is the answer. For robot and human trajectories, $y$ is the physical-language description. This detail matters for interpreting the representation results: the same prediction interface is asked to preserve visual knowledge and to express local manipulation semantics, rather than adding an unrelated classifier head.
The quantization is both a strength and a cost. It gives human and robot data a deterministic common vocabulary, and it prevents the reasoner from drifting into an action-only numerical channel. At the same time, centimeter translation, ten-degree rotation, omitted near-zero dimensions, and binary gripper state erase sub-template motion detail. The dense action flow can still preserve continuous control, but the language path cannot describe fine alignment below those intervals. This limitation becomes visible near contact, where a small positional correction can determine whether a grasp succeeds.
Pretraining Data: Complementary Sources, Conditional Supervision
The pretraining mixture has three categories. VQA data contains about 8.293 million pairs covering spatial understanding, referring expressions, general reasoning, temporal video questions, embodied interaction, and planning. Robot demonstrations come from AgiBot World 2026, AgiBot World Alpha, Bridge, Droid, Fractal, Robocoin, and Interdata-A1. Human egocentric data comes from EgoVerse, EgoDex, and VITRA and totals about 5,072 hours.
| Source | Role in pretraining | Reported scale |
|---|---|---|
| Robot demonstrations | Supervises physical language, future video, and continuous action; includes single-arm, dual-arm, and mobile-platform data | About 4,958 hours in Table 1 and the final scaling point |
| Human egocentric video | Supervises physical language and future visual dynamics; avoids low-precision robot-action supervision | About 5,072 hours |
| VQA and embodied QA | Preserves or develops visual semantics, spatial reasoning, temporal reasoning, and planning | About 8.293 million pairs |
The robot-hour total is not internally consistent. Section 3.1.1 says the selected data total approximately 4,363 hours, while the per-source sum in Table 1 and the last point of the scaling study use about 4,958 hours. This analysis keeps both values. They may refer to different filtering boundaries or an updated table, but the paper does not provide a reconciliation. Reproductions should record the actual trajectory-hour count that enters the optimizer rather than assuming either narrative number is sufficient.
The cleaning pipeline targets control-relevant validity rather than merely visual cleanliness. State and action trajectories are smoothed with cascaded median filtering and Savitzky-Golay regression. A spike is flagged only when its residual, acceleration, and jerk all indicate an anomaly, which avoids deleting genuine fast motion. Shared state-action dimensions are aligned through cross-correlation estimates, and incremental actions are integrated before geometric comparisons. Pose, frame convention, and rotation representation are checked together. Visual filtering removes black, corrupt, severely blurred, or long-static segments while retaining gripper transitions that carry contact information.
Human long videos require segmentation before they can be used as observation-conditioned targets. EgoANT first samples a contact sheet at 0.5-second intervals with timestamps to make a coarse episode segmentation. It then examines local windows around candidate boundaries. Finally, it generates descriptions from raw frames, an alternative video-decoding path, and prior-conditioned candidates, and selects a final action phrase with a text-only model. The result is an interval paired with an action, object, and target-state description.
Figure 2: The three-expert architecture and the data mixture. The paper maps VQA, human, and robot sources to different combinations of language, video, and action supervision. Production asset derived from paper Figure 3.
Joint Flow Matching Objective
The world generator and action predictor use flow matching. Let $Z_t^0$ be clean future visual latents from a frozen video autoencoder and $A_t^0=a_{t+1:t+h}$ be the target action chunk. Let $\hat u_\theta^v$ and $\hat u_\theta^a$ be the predicted video and action velocity fields at noise levels $\tau_v$ and $\tau_a$. The two regression losses can be written as:
$$ \mathcal L_v=\mathbb E\left[ \left\|\hat u_\theta^v-(Z_t^1-Z_t^0)\right\|_2^2 \right], \qquad \mathcal L_{\mathrm{act}}=\mathbb E\left[ \left\|\hat u_\theta^a-(A_t^1-A_t^0)\right\|_2^2 \right]. $$
During pretraining, both sources $Z_t^1$ and $A_t^1$ are Gaussian noise. The visual and action noise levels are sampled independently, producing the interpolants:
$$ Z_t^{\tau_v}=(1-\tau_v)Z_t^0+\tau_vZ_t^1, \qquad A_t^{\tau_a}=(1-\tau_a)A_t^0+\tau_aA_t^1. $$
The current observation latent remains clean and is excluded from the visual loss. The model predicts both velocity fields in one forward pass, allowing the action and video experts to condition on each other while learning their respective targets. The general objective combines the three losses:
$$ \mathcal L =w_a\mathcal L_{\mathrm{act}} +w_v\mathcal L_v +\mathcal L_{\mathrm{lang}}. $$
The appendix resolves the weights used in the main runs as a 1:1:0.1 ratio:
$$ \mathcal L_{\mathrm{pretrain}} =\mathcal L_{\mathrm{video}} +\mathcal L_{\mathrm{action}} +0.1\mathcal L_{\mathrm{language}}. $$
The lower language weight is a practical balance. Text tokens can be long, but video latents and continuous action chunks are dense and numerically large. Weighting language by 0.1 keeps semantic supervision active without allowing token count to dominate the optimization. The total is a multi-objective training problem, not a simple sum of separately normalized model losses.
Post-Training for Imperfect Futures and Short Denoising Paths
Joint attention creates a failure mode. If training always exposes the action expert to a nearly clean future latent, the policy can learn to depend on a future representation that inference cannot reproduce reliably. The sample-level augmentation in the post-training stage therefore perturbs future-frame latents with probability 0.5:
$$ \widetilde Z_t^{\tau_v} =s_{\mathrm{aug}}Z_t^{\tau_v} +(1-s_{\mathrm{aug}})\epsilon_v, \qquad s_{\mathrm{aug}}\sim\mathcal U(0.5,1), \qquad \epsilon_v\sim\mathcal N(0,\mathbf I). $$
No perturbation is applied in the other cases. The conditioning frame remains clean, the flow target is unchanged, and the timestep embedding is not rewritten to match the added noise. This is deliberate. The action expert must recover useful structure from an imperfect visual token stream while still preserving the observed task context. The augmentation changes what attention sees, not what the video generator is asked to denoise.
History-conditioned flow matching changes the action source. Let $A_{t-1}=a_{t-h+1:t}$ be the previous action chunk, which has the same shape as the target $A_t^0$. Instead of starting from Gaussian noise, the source is a perturbed history:
$$ A_t^1=A_{t-1}+\epsilon_a, \qquad \epsilon_a\sim\mathcal N(0,\eta^2\mathbf I), \qquad \eta=0.02. $$
The interpolated action state and velocity target are:
$$ A_t^\tau=(1-\tau)A_t^0+\tau A_t^1, \qquad u_t^a=A_t^1-A_t^0. $$
This starts generation near the motion distribution of the preceding chunk. The model refines continuity and predicts the next chunk instead of rediscovering the current pose, velocity, and gripper rhythm from unrelated noise. The visual source remains Gaussian noise. The paper reports that two to four action-to-action flow steps are sufficient in the tested setting, with four steps used as the default; noise-to-action generation generally needs a longer path.
The two post-training ideas target different uncertainties. Future-noise augmentation makes the action policy less sensitive to errors in imagined visual structure. History initialization reduces optimization distance and improves temporal consistency. The ablation is useful because their contributions are not the same: action-to-action initialization supplies most of the immediate gain, while future-noise augmentation adds a smaller robustness increment.
How the Paper Maps to the Public Implementation
The public repository exposes the architecture directly. In models/motus.py, process_joint_attention creates action and understanding query-key-value tensors, then passes them into the Wan self-attention call together with the video tokens. Separate output projections return each modality to its expert-specific residual stream. This corresponds to the paper's concatenated modality projections and joint attention equation, but the implementation also makes the dimensional adaptation explicit: action width 1,024 and understanding width 2,048 must be projected into the world generator's 24-head attention space.
# models/motus.py:248-324, abridged
def process_joint_attention(
self, video_tokens, action_tokens, video_adaln_modulation,
action_adaln_modulation, layer_idx, action_block,
und_tokens, und_block, und_k_lens=None,
):
"""Trimodal joint self-attention: WAN + Action + Understanding."""
norm_video = wan_layer.norm1(video_tokens) * (1 + v_mod[1]) + v_mod[0]
norm_action = action_block.norm1(action_tokens) * (1 + a_mod[1]) + a_mod[0]
a_qkv = torch.einsum("BTD,KNDE->KBTNE", norm_action,
action_block.wan_action_qkv)
u_qkv = torch.einsum("BTD,KNDE->KBTNE", norm_und,
und_block.wan_und_qkv)
y, action_out_h, und_out_h = wan_layer.self_attn(
norm_video, seq_lens, self.grid_sizes, freqs,
action_q=a_q, action_k=a_k, action_v=a_v,
und_q=u_q, und_k=u_k, und_v=u_v,
)
und_tokens = und_tokens + und_block.wan_und_o(und_out_h)
action_tokens = action_tokens + action_block.wan_action_o(action_out_h)
video_tokens = video_tokens + y * v_mod[2]
return video_tokens, action_tokens, und_tokens
The data and configuration evidence is equally concrete. configs/robotwin_lap_future_noise.yaml uses a 14-dimensional interface, eight future frames, $384\times320$ video, 200,000 optimization steps, learning rate $5\times10^{-5}$, 16-step history, and action-source noise 0.02. utils/video_noise_augmentation.py preserves the conditioning frame, perturbs only future latent frames, and mixes future content according to the sampled retention scale. models/motus.py applies that function after visual interpolation and constructs the action source from history when the configured flow-source mode is history. The implementation confirms that the visual augmentation is an input perturbation rather than an altered training target.
# models/motus.py:928-980 and utils/video_noise_augmentation.py, abridged
video_source = condition_frame_latent.expand_as(clean_full_latent)
noisy_video_latent = clean_full_latent * (1 - sigma) + video_source * sigma
noisy_video_latent = apply_future_frame_noise_augmentation(
noisy_video_latent,
probability=self.config.future_video_noise_aug_probability,
min_scale=self.config.future_video_noise_aug_min_scale,
max_scale=self.config.future_video_noise_aug_max_scale,
)
if self.config.flow_source_mode == "history":
action_source = history_actions + (
torch.randn_like(history_actions)
* self.config.flow_source_action_noise_std
)
else:
action_source = torch.randn_like(actions)
noisy_actions = actions * (1 - sigma_action) + action_source * sigma_action
action_target = action_source - actions
Simulation Results: Strong In-Distribution Control and Smaller Domain Drop
LIBERO tests spatial, object, goal, and long-horizon manipulation under a comparatively fixed visual and linguistic distribution. UniWAM reaches 99.6%, 99.6%, 99.2%, and 98.4%, respectively, for a 99.2% average. This is 0.5 points above the previously best listed Xiaomi-Robotics-0 result. The main conclusion is not that adding a world-generator and a reasoner is free; it is that the joint training recipe avoids sacrificing standard manipulation accuracy.
| Method | Spatial | Object | Goal | Long | Average |
|---|---|---|---|---|---|
| pi0.5 | 98.8 | 98.2 | 98.0 | 92.4 | 96.9 |
| Motus | 96.8 | 99.8 | 96.6 | 97.6 | 97.7 |
| LingBot-VA | 98.5 | 99.6 | 97.2 | 98.5 | 98.5 |
| Xiaomi-Robotics-0 | 98.8 | 100.0 | 98.8 | 97.2 | 98.7 |
| UniWAM | 99.6 | 99.6 | 99.2 | 98.4 | 99.2 |
RoboTwin 2.0 is more relevant to the paper's dual-arm claim. On 50 tasks, UniWAM obtains 75.14% in clean-to-clean (C2C) and 68.32% in clean-to-randomized (C2R), for a 71.73% average. It does not dominate every column: OpenWAM-alpha reaches 89.40% on C2C, well above UniWAM. The important result is robustness to the distribution shift. UniWAM loses 6.82 points from C2C to C2R, while pi0.5 loses 24.70 and Spatial Forcing loses 50.46.
| Method | C2C | C2R | Average | C2C to C2R drop |
|---|---|---|---|---|
| GR00T-N1.7 | 43.60 | 20.70 | 32.20 | 22.90 |
| pi0.5 | 70.70 | 46.00 | 58.35 | 24.70 |
| Spatial Forcing | 77.20 | 26.74 | 51.97 | 50.46 |
| 4D-WAM | 81.50 | 41.80 | 61.70 | 39.70 |
| OpenWAM-alpha | 89.40 | 48.70 | 69.00 | 40.70 |
| UniWAM | 75.14 | 68.32 | 71.73 | 6.82 |
LIBERO-Plus disaggregates robustness across seven shifts. UniWAM reaches 92.6% overall, including 89.5% under robot-initial-state changes, 92.2% under language changes, 97.9% under lighting changes, 97.6% under background changes, 94.3% under sensor noise, 92.0% under camera changes, and 84.6% under layout changes. The camera and layout scores are not the best in the table. The overall lead comes from a broader balance across semantic and visual perturbations rather than from a single dominant dimension.
Figure 3: Real-world task families. The top four groups test object reference, interaction type, spatial relation, and drawer coordination; the bottom sequence is a six-stage desk-organization task. Production asset derived from paper Figures 5 and 6.
Real-World Evidence: Instruction Grounding Is the Sharper Difference
The real-world setup uses two AgileX Piper six-degree-of-freedom arms with parallel grippers and wrist cameras, plus a third-person camera. Each task contributes 200 demonstrations, and one shared policy is trained across tasks. UniWAM, Motus, and pi0.5 are trained for 80,000 steps on eight H100 GPUs. Evaluation covers picking a referenced object, selecting a named interaction, placing according to a spatial relation, storing an object in a drawer, and a six-milestone desk-organization task.
Averaged over the four instruction tasks, UniWAM reaches a 67.5% success rate and an 82.5% instruction-following rate. These are 13.1 and 21.9 points above pi0.5, respectively. The gap is largest on Diverse-Interaction, where the policy must combine a target noun with an intended verb: success rises from 40.0% to 72.5% and instruction following from 50.0% to 80.0%.
| Real-world metric | UniWAM | pi0.5 | Motus | Interpretation |
|---|---|---|---|---|
| Four-task success rate | 67.5% | 54.4% | Not reported as this average | 13.1-point gain in completed tasks |
| Four-task instruction-following rate | 82.5% | 60.6% | Not reported as this average | 21.9-point gain in choosing the requested behavior |
| Diverse-Interaction success | 72.5% | 40.0% | Not reported | 32.5-point gain when noun and verb must be composed |
| Long-horizon progress, maximum 6 | 5.0 | 4.8 | 3.2 | More complete execution under a high-level instruction |
The instruction-following metric separates semantic grounding from raw actuation. UniWAM's advantage is larger on instruction following than on task success, which suggests that it more often selects the correct object and interaction before executing. This interpretation is consistent with physical-language supervision but is not a causal proof by itself. The long-horizon result is also narrower than the headline may suggest: UniWAM's 5.0 is only 0.2 above pi0.5, while the major separation is from Motus at 3.2. The six-milestone score does show that a world-action policy without sufficient semantic grounding can stall between subgoals.
Scaling: Human Video Is Valuable Only After a Threshold
The authors pretrain with 250, 500, 1,000, 2,500, and 5,000 hours of robot data, with and without a matched amount of human egocentric data. Robot-only performance rises from 0.35 average task completion at 250 hours to about 0.67 at 5,000 hours, with no saturation in the tested range. Human data is not uniformly helpful. Below roughly 1,000 robot hours, adding human video reduces performance. From 1,000 hours onward, the effect becomes positive and grows with scale.
The converged validation loss is fitted as a log-linear function of total data hours:
$$ L(D)\approx0.03205-0.00302\ln D, \qquad R^2\approx0.983. $$
$D$ is the number of robot and human pretraining hours, and $L$ is validation mean squared error. The high fit quality is based on a small number of scale points, so it should be read as a useful empirical trend rather than a universal law. Within this study, the validation loss also tracks downstream success, which supports its use as an offline indicator of transfer. The negative-transfer regime is more informative than the fitted line because it identifies a real boundary: embodiment differences dominate until the shared data are large enough for common dynamics to emerge.
Figure 4: Validation loss and RoboTwin clean-to-randomized success as robot data scale increases. The final robot-plus-human point uses 4,958 robot hours and 5,072 human hours. Production asset derived from paper Figure 7.
Representation evidence supports the same story. Without physical-language supervision, robot and human samples occupy largely separate regions in the reasoner's t-SNE projection. With supervision, overlap increases, especially in the central and lower regions. A two-dimensional projection cannot establish that a shared causal mechanism has formed, but it is consistent with the hypothesis that a common action vocabulary reduces the domain gap.
Figure 5: t-SNE of robot and human representations without and with physical-language supervision. The right plot shows greater overlap. Production asset derived from paper Figure 8.
Ablations: What Actually Carries the Gains?
Without pretraining, RoboTwin average success is 46.23%. Robot-only pretraining raises it to 67.53%. Adding human data and VQA produces 71.73%. VQA contributes mainly to randomized evaluation, while human data improves both clean and randomized settings. This pattern matches the intended division of labor, although the aggregate numbers do not isolate the dependence of each gain on scene type.
The reasoner-training ablation is cumulative. A frozen reasoner gives 42.61%; unfreezing raises it to 46.49%; adding physical language gives 51.74%; adding the full pretraining mixture gives 71.34%. The large final step does not mean pretraining alone caused a 19.6-point jump. It means capacity, the physical-language objective, and large-scale data must be present together for the reasoner to become useful to control.
The action-generation ablation is cleaner. Noise-to-action gives 64.80%, history-to-action gives 69.73%, and adding future visual noise gives 71.34%. History initialization supplies the main gain, while future augmentation adds 1.61 points. The paper's appendix also compares success against denoising steps and reports that action-to-action reaches strong performance in two to four steps, whereas noise-to-action remains less efficient.
| Ablation | Configuration | Average success | What changes |
|---|---|---|---|
| Pretraining data | No pretraining | 46.23% | Only downstream demonstrations |
| Pretraining data | Robot data | 67.53% | Adds aligned state-action and video supervision |
| Pretraining data | Robot + human + VQA | 71.73% | Adds human dynamics and preserved visual semantics |
| Reasoner | Frozen | 42.61% | General VLM features are not adapted enough |
| Reasoner | Unfrozen | 46.49% | Adaptation without a strong embodied objective is limited |
| Reasoner | Unfrozen + physical language | 51.74% | Local motion enters the shared language interface |
| Action flow | Noise to action | 64.80% | Long path from an uninformative source |
| Action flow | History to action | 69.73% | Temporal continuity supplies a structured source |
| Video path | History + future noise | 71.34% | Action becomes less dependent on clean imagined futures |
Figure 6: Three ablation groups on RoboTwin 2.0: pretraining mixtures, reasoner training, and action-generation strategies. Production asset derived from paper Figure 9.
Reproduction Boundary and Limitations
The first limitation is dependence on future visual quality. Future-noise augmentation teaches the action expert to tolerate imperfect latents, but it does not establish robustness to a world generator that loses object identity or contact geometry. The paper does not systematically evaluate severe occlusion, rapid camera motion, transparent or reflective objects, or scenes outside the video backbone's training distribution. If the future tokens become semantically wrong rather than merely noisy, repeated joint attention can propagate the error into control.
The second limitation is scale-dependent human transfer. The paper directly reports negative transfer below about 1,000 robot hours. This rules out the simple claim that adding human video always improves a robot policy. Hand appearance, camera geometry, workspace calibration, motion precision, and embodiment differences can dominate at small scale. The threshold is also tied to this data mixture and physical-language template; it should not be treated as a platform-independent constant.
The third limitation is quantization in the physical-language interface. Centimeter translation, ten-degree rotation, omitted unchanged dimensions, and a binary gripper state are useful for cross-embodiment alignment, but they omit sub-centimeter and sub-ten-degree corrections. Dense action flow remains continuous, yet the language specialist cannot describe fine insertion, gentle contact, or subtle compliance. This may explain why the paper's strongest evidence is tabletop manipulation rather than precision assembly.
The fourth limitation is embodiment coverage. The real-world experiments use one AgileX Piper dual-arm platform, 200 demonstrations per task, and 40 trials per condition. The instruction-grounding gains are meaningful, but they do not establish transfer to mobile manipulation, dexterous hands, deformable objects, high-force assembly, or multi-robot collaboration. The simulation suites are broader in task count, but they still do not replace real-platform diversity.
The fifth limitation is reproducibility metadata. Pretraining uses seven days on 32 H800 GPUs. The paper says RoboTwin post-training uses eight H800 GPUs for 12 hours, while the public README says eight H100 GPUs for about 10 hours. These may reflect different software stacks or hardware revisions, but the discrepancy is not reconciled. The public release includes the base model, RoboTwin checkpoints, LIBERO training and evaluation code, but deliberately excludes Bridge, DROID, Fractal, and real-world deployment from its current scope. Full data-pipeline and physical-robot reproduction is therefore not available from the repository alone.
There is also a sixth, smaller inconsistency: the robot pretraining total is about 4,363 hours in Section 3.1.1 but about 4,958 hours in Table 1 and the final scaling point. The benchmark comparisons are unaffected, but data-budget reproduction and scaling-law interpretation require the actual optimized-hour count.
Conclusion
UniWAM's central contribution is not merely that it combines three large backbones. It makes the relation among understanding, future imagination, and action a repeated computation inside a 30-layer decoder. Physical language gives heterogeneous human and robot trajectories a shared target vocabulary; joint attention lets action tokens query semantic and visual tokens before the future is finalized; and post-training aligns flow matching with noisy future representations and temporally continuous control.
The evidence supports three narrower claims. The architecture retains strong LIBERO performance at a 99.2% average. The unified training recipe reduces the RoboTwin C2C-to-C2R drop to 6.82 points, far below several baselines. Human video helps only after the data scale is large enough, with negative transfer below roughly 1,000 robot hours. Those results are more credible than a claim of universal embodiment transfer because they include counterexamples, per-dimension failures, and scale dependence.
The unresolved question is whether the shared representation actually transfers causal dynamics across embodiments, or whether it mainly supplies a useful regularizer at large scale. Future work needs controlled tests on new robot embodiments, precision contact tasks, and world-generator failures severe enough to break object relations. A useful next benchmark would keep the language and action interfaces fixed while replacing the video backbone with a deliberately weaker one. If performance collapses, future visual reasoning is essential; if it remains stable, the current gains may come more from language and temporal action history than from imagined video.
UniWAM's real test is not whether it can generate a plausible future, but whether an action expert learns what remains trustworthy when that future is wrong.
