PAPER DEEP DIVE
EWAM: Emergent Depth-Wise Specialization in a Unified Embodied Model – From Semantic Understanding through Visual Foresight to Action
Vision-language-action (VLA) policies emphasize semantic understanding, whereas world-action models (WAMs) learn predictive representations of environment dynamics. Systems that expose a policy to both sources often still concentrate action computation on a single expert. We present EWAM, an action-centric unified embodied model whose asymmetric joint attention lets action tokens read semantic, current-visual, predicted-future, and action information at every layer while the perceptual experts retain their distinct roles. Without layer-wise supervision, EWAM develops an emergent depth-wise specialization: action queries attend mainly to vision-language features in shallow layers, to predicted future frames in intermediate layers, and to action tokens themselves in deep layers. This handoff replicates across tasks and is stable across denoising steps. Checkpoint tracking and causal interventions show that it is learned and that action generation depends on it. EWAM is pretrained in two separate regimes, one on cross-embodiment robot trajectories and one on human egocentric video. In simulation and real-robot experiments, it surpasses existing VLA, WAM, and hybrid baselines. Human egocentric data improve both cross-embodiment transfer and real-robot robustness, and subtask-phase supervision improves long-horizon completion. Together, these results suggest that unified embodied learning can induce an ordered internal progression from semantic understanding, through visual foresight, to action formation.
One-Sentence Summary
EWAM places vision-language understanding, video prediction, and action generation in one depth-aligned transformer stack, lets action tokens alone read the other experts, and discovers without layer supervision that action queries learn a reliable progression from semantics in shallow layers, to predicted futures in middle layers, to action refinement in deep layers.
Background and Motivation
A general robot policy must understand an instruction and the current scene, anticipate how the physical world may change, and convert those signals into precise control. Vision-language-action policies inherit strong semantics from pretrained vision-language models, while world-action models learn physical dynamics from video. Each paradigm supplies an important resource, but a manipulation task generally needs both. The unresolved question is not whether two capabilities can be attached to one system. It is how action computation should coordinate them internally.
One common strategy is to add an auxiliary future-image or latent-prediction objective to a VLA. Another is to run a world model first and pass a future frame or subgoal into a downstream policy. These designs can improve capability, but they often leave action generation concentrated on one dominant stream. A separately generated future also introduces a hard interface: errors in the generated subgoal can propagate into control, and the action model may not receive useful predictions until generation is complete.
EWAM asks a sharper question: when semantic, visual-future, and action tokens are jointly available, does the action query use one source everywhere, or does its source preference change with depth? The authors hypothesize that a properly unified model may learn an ordered internal progression. Shallow computation identifies the task and grounds the current scene. Intermediate computation shifts visual dependence toward predicted scene evolution. Deep computation refines the motor trajectory using concise semantic and predictive context.
The claim is tested in four ways. First, action-attention allocation is measured across depth. Second, the pattern is checked across all 50 RoboTwin 2.0 tasks and across denoising steps. Third, checkpoint tracking asks whether the pattern emerges during a single training run. Fourth, layer-targeted masking and counterfactual future-feature injection test whether the selected sources are functionally necessary and whether their content changes behavior. This combination is what distinguishes EWAM from a paper that merely visualizes attention heatmaps.
Problem Formulation and Preliminaries
Given a current observation $o_t$ and instruction $\ell$, the policy predicts a future visual trajectory $v_{t+1:t+H}$ and an action chunk $a_{t:t+K-1}$. The predicted future is not an external prompt. Its latent features remain available as keys and values to action queries throughout the network. Both video and action targets use flow matching. For a clean target $x$ and Gaussian noise $\epsilon$, the model constructs a noised interpolation and regresses the velocity from noise toward data:
$$x_\tau=(1-\tau)x+\tau\epsilon,\qquad u^\star(x_\tau,\tau)=\epsilon-x.$$
Video and action noise levels are sampled independently. If both streams shared the same noise schedule, the network could exploit the known video corruption level as a shortcut for estimating the action target. Independent schedules prevent that coupling and force the action stream to extract useful information from the predicted video representation itself.
The architecture contains three depth-aligned experts. A Qwen3-VL-2B vision-language expert encodes the instruction and current image. A Wan2.2-TI2V-5B video expert processes the condition-frame latent and noisy future-video latents. A lighter action expert embeds the robot state and noisy action chunk. Their corresponding blocks execute at the same depth, but the three experts are not allowed to communicate freely.
Method
Action-Centric Asymmetric Attention
The action stream is the only integration interface. Let $V$, $A$, and $L$ denote video, action, and vision-language tokens. The attention mask permits action queries to see every source, but keeps each perceptual expert within its own stream:
$$A\leftarrow\{V,A,L\},\qquad V\leftarrow V,\qquad L\leftarrow L.$$
This asymmetry protects the roles learned during pretraining. The video expert is not rewritten by language tokens, and the vision-language expert is not directly shaped into a controller by the action loss. They influence control only through the keys and values they expose to action queries. That separation also creates a well-defined object of analysis: action-query attention shows exactly which source is being retrieved at each layer.
The implementation evaluates the three query groups with one concatenated masked-attention operation. Each stream still has its own normalization, projection, and output parameters, but a single kernel processes the joint token sequence. The mask is not merely a training detail. It defines the information topology of the model and is the reason the paper can later mask one source at one depth without changing any weights.
Qwen3-VL produces 28 hidden states, while the unified stack has 30 blocks. EWAM aligns the sequence by keeping the first 28 states and repeating states 26 and 27 at the end. At block $l$, the aligned vision-language feature $q_l^L$ is fused with the accumulated stream state $z_{l-1}^L$:
$$x_l^L=\begin{cases}q_l^L,&l=0,\\[2pt]\frac{1}{2}\left(q_l^L+z_{l-1}^L\right),&l>0.\end{cases}$$
The fusion preserves the native Qwen3-VL computation at every depth while allowing action-driven residual updates to alter the semantic keys visible to later blocks. No extra feed-forward network is added on the VL side because each $q_l^L$ already includes the original model block's feed-forward result.
Joint Video-Action Flow Matching
The current frame is encoded once as a fixed condition latent. Noisy future-video latents and noisy action chunks are initialized separately. During training, the video expert predicts a future-latent velocity and the action expert predicts an action velocity in the same forward pass. The base objective is:
$$\mathcal{L}_{\mathrm{base}}=\lambda_v\left\lVert\hat{u}_v-u_v^\star\right\rVert_2^2+\lambda_a\left\lVert\hat{u}_a-u_a^\star\right\rVert_2^2.$$
Here $\hat{u}_v$ and $\hat{u}_a$ are predicted velocities, while $u_v^\star$ and $u_a^\star$ are the target flow fields. At inference, both trajectories are integrated together. Action queries therefore see evolving predicted-future features at every denoising step rather than waiting for a completed future video. This is the central architectural difference from a pipeline that first generates a visual subgoal and then invokes a policy.
Figure 1: EWAM aligns a vision-language expert, a video expert, and an action expert across depth. Action tokens are the only stream allowed to read all three token sources.
How the Mask Is Implemented
The repository function models/ewam.py::process_joint_attention makes the paper's diagram concrete. Video, action, and vision-language tokens are projected into a shared 24-head space with head width 128. The code creates a boolean mask in which video query rows can see only video key columns, VL query rows can see only VL columns, and action rows can see all columns. The three query groups then enter one masked-attention call and return through separate output projections and residuals.
The action expert does not own a separate cross-modal attention kernel. $ActionExpertBlock$ provides action-side QKV projections, normalization, output projection, and feed-forward layers, while the video backbone executes the joint attention. This matches the paper's design principle: perceptual experts maintain independent computation, and the action stream performs integration.
Measuring Where Action Queries Look
For every action query and visible key, the authors compute post-softmax attention weights. Let $\mathcal{Q}_A$ be the action-query set, $\mathcal{K}_m$ the keys from source $m$, and $\alpha_{qk}^{(l)}$ the weight from query $q$ to key $k$ at layer $l$. The source allocation is:
$$R_l^m=\frac{\sum_{q\in\mathcal{Q}_A}\sum_{k\in\mathcal{K}_m}\alpha_{qk}^{(l)}}{\sum_{q\in\mathcal{Q}_A}\sum_{k\in\mathcal{K}}\alpha_{qk}^{(l)}}.$$
The analysis separates five sources: VL text, VL image, observed condition frame, predicted future frame, and action tokens. Because source key counts differ, the paper also reports token-count-normalized enrichment:
$$E_l^m=\frac{R_l^m}{|\mathcal{K}_m|/\sum_j|\mathcal{K}_j|}.$$
Values above one mean that a source receives more attention than it would under uniform allocation across all keys. This normalized view matters because future-frame keys are numerous. Raw attention mass alone can confuse key abundance with functional importance.
Understand, Foresee, Act
Across one representative sequence from each of 50 RoboTwin tasks, each task first averages the five measured denoising steps and the paper then macro-averages across tasks. In layers 0 to 9, VL tokens receive 75.4% of action attention, video receives 14.1%, and action self-attention receives 10.6%. The shallow network is primarily grounding the instruction and current scene.
In layers 10 to 19, the allocation becomes 36.6% VL, 24.3% video, and 39.1% action. The visual handoff is concentrated in layers 11 to 15, with a median crossover from VL-image attention to future-frame attention at layer 13 and an interquartile range of 12 to 13. Fine-grained curves show that image attention drops while predicted-future attention rises. The observed condition frame remains relatively weak. The model is not merely reading more video; it is shifting toward a prediction of how the scene will evolve.
In layers 20 to 29, action self-attention rises to 59.5%, while VL and video fall to 25.2% and 15.3%. Averaged over layers 16 to 29, VL-image receives only 1.3% of action attention and predicted future receives 11.8%. The deep stream preserves compact semantic and predictive context but spends most of its computation refining the motor trajectory. The three regimes overlap, so the result is a graded handoff rather than three hard-coded modules.
Figure 2: VLA, WAM, and multi-expert policies concentrate action attention on one source. EWAM changes its dominant source with depth.
Does the Handoff Emerge During Training?
The authors track twelve checkpoints from one 32-GPU run, spanning 50 to 40,000 optimizer steps. At step 50, action queries attend to VL-image tokens at nearly every depth, and VL-text receives only 2.9% of shallow attention. By step 150, shallow text attention reaches 22.6%, image attention concentrates in shallow layers, and future-frame raw attention first exceeds VL-image mass at layer 8. At step 40,000, shallow grounding, intermediate future prediction, and deep action formation are established.
The mask specifies what each stream may access, but not where a source should be used. The depth-specific routing therefore emerges from training. The trajectory is not monotonic: under token normalization, shallow image enrichment rebounds to 1.79 at 5,000 steps before deep image enrichment falls to 0.04 at convergence. The model passes through a period of representation reorganization rather than following a strictly linear schedule.
Figure 3: The future-frame advantage over VL-image attention expands toward shallower layers during training until the intermediate visual-foresight region stabilizes.
Causal Evidence From Masking
Attention plots alone cannot establish that a source matters. The paper first trains structural variants without pretraining, then applies inference-time masks to one fixed robot-pretrained checkpoint. On the fixed model, the full policy reaches 92.80% success on RoboTwin Randomized. Masking all video keys in layers 0 to 9 changes performance only to 91.98%, but masking VL keys in the same layers drops it to 76.53%. Shallow action computation therefore depends much more on semantic grounding than on visual prediction.
The middle layers show the opposite pattern. Masking the condition frame alone retains 81.40%, but masking all video keys leaves only 0.25%, and masking future-frame keys alone collapses success to 3.55%. The critical middle-layer signal is predicted future content, not visual tokens in general. Masking all video keys in layers 20 to 29 reaches 65.82%, showing that deep action refinement still uses future information but depends on it less than the middle stage.
| Fixed-checkpoint intervention | RoboTwin Randomized success | Change |
|---|---|---|
| Full EWAM | 92.80% | Baseline |
| Mask VL keys, layers 0 to 9 | 76.53% | -16.27 points |
| Mask condition-frame keys, layers 10 to 19 | 81.40% | -11.40 points |
| Mask future-frame keys, layers 10 to 19 | 3.55% | -89.25 points |
| Mask all video keys, layers 10 to 19 | 0.25% | -92.55 points |
| Mask all video keys, layers 20 to 29 | 65.82% | -26.98 points |
The authors also run a content-sensitive counterfactual test. They cache future-frame keys and values from a successful donor rollout in layers 10 to 19 and inject them into five target rollouts for the first three policy calls. The target condition frame, VL features, state, and action representation are preserved. Across five tasks, success falls from 25/25 without injection to 4/25 with injection. In the bottle-adjustment task, injected targets initially follow the donor's arm selection despite different bottle identity, orientation, and position. Some recover after injection stops. This shows that intermediate future features carry control-relevant spatial and action information rather than acting as a decorative auxiliary pathway.
A Shared Action Space for Human Video
Human egocentric video does not contain robot joints or gripper commands. EWAM instead expresses wrist motion in the anchor camera frame. Given wrist poses $(R_{we},t_{we})$ and $(R_{we^\star},t_{we^\star})$ plus camera orientation $R_{wc}$, the motion is first expressed in the wrist frame and then mapped through $R_{ce}=R_{wc}^{\top}R_{we}$:
$$\Delta R_c=R_{ce}R_{ee^\star}R_{ce}^{\top},\qquad \Delta t_c=R_{ce}t_{ee^\star},\qquad \Delta\omega_c=\log_{SO(3)}(\Delta R_c).$$
Each hand contributes translation and rotation, plus two reserved slots, producing a 16-dimensional action:
$$a=\left[\Delta t_c^L,\Delta\omega_c^L,0,0,\Delta t_c^R,\Delta\omega_c^R,0,0\right].$$
On the robot side, the reserved slots carry gripper commands. Every action in a chunk is referenced to the same anchor pose and camera, so action $i$ is the displacement from the anchor to frame $i+1$, rather than a frame-to-frame increment. This removes world-origin and axis conventions while preserving metric translation scale. It does not eliminate kinematic differences between a human wrist and a robot end effector.
Figure 4: Egocentric fisheye video, camera pose, and two 6-DoF wrist trackers are reduced to camera-relative wrist actions that can be placed in the robot action layout.
Subtask and Phase Supervision
Long-horizon demonstrations are divided into $N$ semantic phases. If the current phase is $k_t$, its normalized index is:
$$p_t=\frac{k_t}{\max(N-1,1)}.$$
The predicted phase index is not trajectory time. It is a position in the annotated subtask sequence. A lightweight autoregressive decoder predicts the text of the active subtask from the final action representation and aligned VL representation:
$$\mathrm{Mem}_t=\left[W_Ah_t^A;W_Lh_t^L\right].$$
Its token probability factorizes as:
$$p_\theta(y_t\mid\mathrm{Mem}_t)=\prod_{j=1}^{M}p_\theta(y_{t,j}\mid y_{t,<j},\mathrm{Mem}_t).$$
A two-layer MLP regresses the phase index from mean-pooled action features. The post-training objective is:
$$\mathcal{L}_{\mathrm{post}}=\mathcal{L}_{\mathrm{base}}+\lambda_p\mathcal{L}_{\mathrm{prog}}+\lambda_s\mathcal{L}_{\mathrm{sub}}.$$
Both additional losses use the same phase partition and no external high-level planner is required. The benefit is largest in long sequences: instructed ranking and stacking rises from 1.5% to 16.0% complete-episode success, even though the absolute result remains modest. The model is being trained to recognize and finish the active phase, not to call a separate planner.
Results
EWAM is evaluated on RoboTwin 2.0, LIBERO, cross-embodiment transfer, and real robots. In the clean-to-random protocol, a single policy is adapted on clean demonstrations and tested under domain randomization. EWAM reaches 82.2% clean and 72.1% randomized success, averaging 77.2%. This exceeds GigaBrain-0.7 at 67.3%, π0.5 at 58.35%, and Fast-WAM at 39.85%.
| Model | Paradigm | RoboTwin Clean | RoboTwin Randomized | Average |
|---|---|---|---|---|
| π0.5 | VLA | 70.7% | 46.0% | 58.35% |
| Fast-WAM | WAM | 77.8% | 1.9% | 39.85% |
| 4D-WAM | WAM | 81.5% | 41.8% | 61.6% |
| GigaBrain-0.7 | VLA + WAM | 66.8% | 67.9% | 67.3% |
| EWAM | Unified | 82.2% | 72.1% | 77.2% |
In the in-domain setting, every method is trained on clean and randomized demonstrations. EWAM reaches 93.0% clean, 92.8% randomized, and 92.9% average. The 0.2-point clean-to-random gap indicates stable operation rather than a narrow dependence on training conditions. On LIBERO, EWAM scores 98.6% Spatial, 99.8% Object, 98.6% Goal, and 98.2% Long, for 98.8% overall.
Real-world evaluation covers seven tasks on a Franka single arm, a Dobot dual-arm platform, and a Unitree G1-D humanoid. EWAM is the highest-scoring entry on every task. It improves Stack Bowls from 70% to 80% and Object into Box from 80% to 90% relative to π0.5. On G1-D it raises Pour Beans from 60% to 80%. The task set includes rearrangement, deformable-object handling, and pouring, so the result is not confined to a single manipulation family.
Human Data, Transfer, and Robustness
Human pretraining uses approximately 2,084 hours of public egocentric video. Robot pretraining uses about 1,800 hours and 300,000 trajectories. On held-out Aloha-Agilex-2, human-pretrained initialization improves success from 36.2% to 66.9% at 120K robot-training steps. Comparing each policy's best evaluated checkpoint gives 40.1% versus 66.9%.
On real G1-D pouring, two policies share 60 teleoperated robot episodes and 30 retargeted human episodes but differ in initialization. Robot pretraining reaches 40% success; human pretraining reaches 60%. Training loss falls from 0.068 to 0.034, and held-out action L2 falls from 0.0231 to 0.0164. A robot-only 90-episode reference reaches 70%, but it changes the data mixture and does not isolate the contribution of human data.
The cleanest data-scaling test fixes 60 robot episodes and adds only egocentric episodes. Under tablecloth and cup changes, real-robot success rises from 10% with no human data to 20% with 30 episodes and 60% with 1,000 episodes, without saturation. Thirty human episodes already match the robustness of a 90-episode robot-only run, which is significant given the cost of collecting teleoperation data.
Real-Time Inference
On one RTX 5090, EWAM generates a chunk of 16 actions with 16 dimensions in 258 ms at five flow-integration steps. Future-frame VAE decoding is omitted because control needs only the action output; the video expert still runs and its latent future features remain available to action queries. Graph-compatible rotary position embeddings and full-graph compilation reduce ten-step latency from 1,668 ms to 461 ms, or 464 ms with chunk-overlap correction. Five-step latency is 258 ms. This is about 12% of the 2.13-second nominal action horizon, leaving room for real-time chunking where the next chunk is generated while the current one executes.
Paper-to-Code Correspondence
The public repository matches the technical report rather than serving only as a model card. In models/ewam.py, the Ewam class initializes the Wan2.2 video model, the Qwen3-VL module, and the action expert. Its process_joint_attention function explicitly builds the asymmetric boolean mask and sends all three QKV groups into one attention call.
In models/action_expert.py, StateActionEncoder maps the robot state and action chunk through multilayer SiLU projectors and adds sinusoidal position embeddings. ActionExpertBlock projects action features into Wan's 24-head attention space. In models/ewam.py::training_step, video and action noise are sampled independently, their flow targets are computed separately, and padded action dimensions are excluded from the action loss. The inference function inference_step jointly denoises both modalities, while computing the Qwen3-VL layer features once outside the loop because they do not depend on the denoising state.
Limitations
The attention analysis shows where action queries retrieve information, and the interventions demonstrate functional dependence, but neither gives a complete causal decomposition of every learned representation. Non-linear compensation across sources may hide effects that are not visible in attention weights or single-source masks. Representation probing, activation patching, and cross-layer mediation would provide a stronger account of what each source contributes.
Three large experts remain computationally expensive. The reported 258 ms latency is measured on one RTX 5090 at batch size one and should not be treated as an embedded-controller result. The video expert must still compute latent futures even when decoded frames are skipped.
Camera-relative wrist motion removes world-frame origin dependence and preserves metric scale, but it does not remove kinematic differences between human wrists and robot end effectors. Calibration and tracking errors can still enter the shared action representation. The real-robot robustness study varies cups, placement, background, and clutter; it does not cover open-world lighting, severe occlusion, tool use, or long-term autonomous operation.
Subtask and phase supervision depends on annotated transition checkpoints. It avoids an external planner, but it requires a semantic phase definition for each long-horizon task. Incorrect phase boundaries can teach the model the wrong progress signal.
Conclusion and Outlook
EWAM's main contribution is a measurable and testable organization principle for unified policies. Action tokens are the only readers of semantic and predictive representations. Training then discovers that the action stream does different work at different depths: shallow layers ground the instruction and scene, middle layers exploit predicted futures, and deep layers refine motion. The pattern replicates across 50 tasks, remains stable across denoising steps, appears during training, and has causal support from masking and counterfactual injection.
The natural next step is adaptive routing. Ambiguous instructions may need longer semantic grounding, uncertain contact may require more predictive computation, and well-specified motion may permit an earlier shift into action refinement. A second direction is counterfactual planning: generate multiple action-conditioned futures and compare their consequences before committing, turning visual foresight from a single predicted trajectory into an internal evaluation mechanism.
Golden Line
Unification is not placing three experts in one diagram; it is letting action queries find the right information at the right depth.
flowchart LR
X["Image and instruction"] --> L["Vision-language expert
self-attention only"]
O["Condition frame and noisy future latents"] --> V["Video expert
self-attention only"]
A0["State and noisy action chunk"] --> A["Action expert"]
L --> J["Action-query asymmetric joint attention"]
V --> J
A --> J
J --> A1["Predicted action chunk"]
J --> P["Phase progress"]
J --> S["Active subtask"]
V --> F["Predicted future video"]
