AW-0: Planning With Code Inside a Joint World Action Model

Robot policies have recently converged on a powerful idea: predict a future visual state, then use that prediction to guide an action. Code-as-policy systems pursue another idea: represent a task as explicit, reusable programs. Both approaches offer something the other lacks. A world action model is grounded in physical dynamics but struggles to expose a reusable plan. A coding agent can compose skills and reason symbolically, but its calls are only as good as its perception and are not inherently constrained by a learned model of what will happen next.

AW-0, short for Agentic World Action Model, makes the connection unusually literal. At each replanning point it jointly denoises three streams: a short skill program, a future video latent, and an action chunk. The code expert reads the current images and instruction; the video expert reads the code; the action expert reads code and future visual features. The three streams are not separate modules stitched together after training. They occupy different token spaces but exchange information through joint attention at every layer of a Mixture-of-Transformers backbone.

The work comes from UC Berkeley, NVIDIA, and the University of Pennsylvania. The authors evaluate on RoboTwin 2.0 and eight real-world bimanual tasks, spanning parallel grippers and 22-DoF dexterous hands. Their strongest average result is not a claim that code always beats direct policy execution. It is a more specific claim: generated code can act as an intermediate semantic representation that improves prediction and control when it is trained on temporally aligned programs and coupled to video and action inside the same denoising process.

Why separate planning and world modeling is a bottleneck

A conventional world action model answers a compact question: given the current observation, instruction, and state, what future visual state and action should follow? This is attractive because the future-video objective can absorb visual dynamics that are hard to specify in a reward or a symbolic planner. It also has a limitation. If a task requires composing several known routines, the model has no explicit place to write down that composition. Skill reuse is hidden inside a high-dimensional policy rather than represented as a program.

Code-as-policy takes the opposite route. APIs for perception and control become vocabulary; a VLM writes a sequence such as locating an object, approaching it, closing a gripper, and placing it. The resulting program is inspectable and compositional, but it does not by itself learn a fine-grained model of future pixels. A wrong object binding or a stale contact assumption can survive until runtime because the symbolic call looks syntactically reasonable.

AW-0 places both representations in one generative process. For the current context $ \mathcal{C}_t=\{\mathcal{O}_t,L,S_t,M_t\} $, containing multi-view observations, the language instruction, robot state, and a compact program memory, it models the joint conditional distribution

$$ p_\theta(C_0,Z_0,A_0\mid\mathcal{C}_t), $$

where $C_0=(c_1,\ldots,c_{N_c})$ is a discrete program token sequence, $Z_0$ is a continuous video latent, and $A_0=(a_t,\ldots,a_{t+H-1})$ is an action chunk of horizon $H$. This equation matters because it does not define code as a post-hoc explanation or a text prompt passed to a frozen controller. Code is a target random variable inside the model.

Three experts, one attention space

The backbone is a Mixture-of-Transformers with separate code, video, and action experts. Each expert has its own layer normalization, feed-forward network, and attention projections, but all three communicate through masked joint attention in each of 28 layers. Attention is bidirectional inside an expert and blockwise causal across modalities in the order $C\rightarrow V\rightarrow A$. Code can therefore organize video prediction, while video features can inform action without allowing future action tokens to leak backward into the plan.

Stream	Main initialization	Training signal	Role at inference
Code	SDAR-1.7B body with frozen tokenizer-side components	Masked-token cross-entropy on temporally aligned program slices	Writes a short skill plan that conditions the other streams
Video	Cosmos-Predict2.5-2B video DiT and frozen Wan2.2 VAE	Flow matching on future video latents	Produces a predictive visual representation of near-future dynamics
Action	Separate 28-layer action DiT with 1,024-wide residual stream	Flow matching on the current action chunk	Produces the action chunk that drives the robot in default mode

The code expert does not look only at text. A frozen Qwen3-VL-2B vision tower, with a trainable merger, encodes three camera views at 224 by 320 pixels. The resulting 210 visual tokens are prepended to instruction and program tokens. The video and action experts use language embeddings through cross-attention, and the action expert additionally receives robot state. The shared 2,048-dimensional attention space is the bridge: each modality retains its own expert computation while code, video, and action remain mutually visible during generation.

flowchart LR
    O["3 camera views
224 x 320"] --> VIT["Frozen Qwen3-VL-2B tower
+ trainable merger"]
    L["Language instruction"] --> C["Code expert
SDAR-1.7B body"]
    VIT --> C
    M["Program memory
previous slices"] --> C
    C -->|"C to V"| V["Video expert
Cosmos-Predict2.5-2B DiT"]
    O --> V
    C -->|"C to A"| A["Action expert
28-layer action DiT"]
    V -->|"V to A"| A
    S["Robot state"] --> A
    C --> P["Generated skill program"]
    V --> F["Imagined future video"]
    A --> ACT["Action chunk"]

Joint denoising with two different corruption processes

Video and action are continuous, while code is discrete. AW-0 therefore uses flow matching for the former and absorbing-mask diffusion for the latter. At a denoising step with masked code $\widetilde C_r$, noisy video latent $Z_{\sigma_v}$, and noisy action chunk $A_{\sigma_a}$, the network predicts

$$ (\ell_\theta^c,v_\theta^v,v_\theta^a) =F_\theta(\widetilde C_r,Z_{\sigma_v},A_{\sigma_a}, \sigma_v,\sigma_a;\mathcal{C}_t). $$

Here $r$ is the code masking ratio, $\sigma_v$ and $\sigma_a$ are noise levels, $\ell_\theta^c$ are code-token logits, and the two velocity fields predict how to move the continuous samples toward the clean video latent and action chunk. The three predictions arise from one forward pass, but each modality keeps a method appropriate to its data type.

For a continuous modality $m\in\{v,a\}$, with clean value $X_0^m$, AW-0 constructs a noisy sample and regresses a velocity target:

$$ X_{\sigma_m}^m=(1-\sigma_m)X_0^m+\sigma_m\epsilon_m, \qquad \epsilon_m\sim\mathcal{N}(0,I), \qquad \epsilon_m-X_0^m. $$

Video uses a shifted sampling distribution $u_v=sb$, where $b\sim\mathrm{Beta}(7,1)$ and $s=0.999$, while action noise uses $u_a\sim U(0,1)$. A shift of $\gamma_m=5$ concentrates samples where they are useful for the pretrained video model and the action head.

Code is corrupted differently. For each token, AW-0 samples a mask ratio $r\sim U(0,1)$ and independently replaces a token with the absorbing mask:

$$ \widetilde c_i= \begin{cases} [\mathrm{MASK}], & b_i=1,\\ c_i, & b_i=0, \end{cases} \qquad b_i\sim\mathrm{Bernoulli}(r). $$

Only the active program slice is corrupted and supervised. The observation prefix, instruction, and memory remain visible. At inference, the policy appends a canvas of $G$ mask tokens and commits the $G/T$ most confident positions at each of $T$ denoising steps. This confidence-ordered masked decoding produces a short program without generating it strictly left to right.

Making video prediction useful to action

Joint training alone does not guarantee that the video stream exposes the right information when the action expert needs it. The video expert inherits a pretrained high-noise distribution that emphasizes broad visual structure. AW-0 mixes that distribution with a uniform reference distribution,

$$ u_v\sim q(u)=\alpha p_{\mathrm{pre}}(u)+(1-\alpha)p_{\mathrm{ref}}(u), \qquad p_{\mathrm{ref}}=U(0,1), $$

with $\alpha=0.5$. It then reweights the modality losses by importance ratios and trains the combined objective

$$ \mathcal{L}=\mathbb{E}_{u_v\sim q}\left[ \lambda_v\bar w_v\ell_v+ \bar w_g(\lambda_a\ell_a+\lambda_c\ell_c) \right], $$

where $w_v=p_{\mathrm{pre}}/q$, $w_g=p_{\mathrm{ref}}/q$, bars denote clipped weights normalized to unit batch mean, $\ell_v$ and $\ell_a$ are flow-matching losses, and $\ell_c$ is masked-code cross-entropy. The practical goal is predictive guidance: sample video at noise levels that teach visual features useful for action, without abandoning the pretrained video distribution.

Where 2.09 million aligned programs come from

Code supervision would be useless if “code” meant arbitrary text written after watching a success video. AW-0’s annotation pipeline constrains programs to a declaration-only API and aligns each program slice with the trajectory segment it explains. Robot pretraining contains 73,101 YAM episodes, 279 tasks, and 1,919.30 hours. Egocentric human manipulation contributes 175,541 episodes, 327 tasks, and 993.91 hours. Together they form 248,642 episodes, 2,913.22 hours, and 2,075,373 subtask slices. Pretraining samples roughly 2,000 hours of YAM data and 1,000 hours of human data.

Property	YAM robot data	Egocentric human data
Episodes	73,101	175,541
Tasks	279	327
Duration	1,919.30 hours	993.91 hours
Frozen skill library	23 skills	27 skills
Verification signal	Gripper contact event sequence	Per-hand motion-path evidence

The pipeline has three stages. First, Gemini 3.1 Pro writes subtask-level programs for a sample of episodes against APIs that declare what each call would do, while Claude Opus 4.8 running through Claude Code acts as a coordinator that distills recurring patterns into general skills. Second, the skill library is frozen and the sample is annotated again with the final vocabulary, producing cached program variants. Third, the rest of the corpus reuses a cached variant when its contact and motion evidence matches; only cache misses trigger another VLM annotation.

For a parallel-gripper robot, a program’s declared contact events are compared with the observed gripper-event sequence. For egocentric human video, the system uses the motion profile of each hand and finds the nearest verified variant. This is more than syntactic parsing. It makes the annotation pipeline sensitive to when contact occurs and how the hand moves, while avoiding an impossible requirement to execute every declared API call on historical teleoperation data.

The reporting needs one important qualification. Matching contact events and hand trajectories verifies alignment; it does not prove that every object argument, count, or spatial relation is correct. A program can contain the right primitive sequence while binding the wrong instance. AW-0’s code loss teaches the model to reproduce the annotation distribution, so annotation errors and gaps in the 50-skill library remain part of the system’s error budget.

Default mode: code guides action but is not executed

This is the most important operational detail in AW-0. In the default policy, the generated program is not sent to a code interpreter or robot API. It is a symbolic plan that conditions future-video and action generation. The action chunk from the action expert drives the robot. Calling AW-0 “agentic” therefore does not imply that every inference step runs generated Python; it means that planning code is a first-class learned representation inside the world action model.

The default design keeps the safety and continuity burden on the learned action channel while exploiting code for semantic structure. Code can express a sequence such as finding, grasping, moving, and placing. The video stream predicts how the scene should evolve under that plan. The action expert converts the combined representation into continuous commands. Because code is represented as a short slice rather than a full episode script, a compact memory of previous slices supplies progress context.

AW-0 also offers a hybrid execution mode. Here a program may become executable after gates for the current stage, prerequisites, syntax, allowed primitives and helpers, and non-vacuity. If the program is attempted, the same-pass action chunk is discarded and the environment executes the generated calls through a bound perception and control API. If execution raises an exception, the system replans immediately. In compact form, the channel choice at replanning time is

$$ a_t= \mathbf{1}[g_t=1]\,a_t^{\mathrm{code}} +\mathbf{1}[g_t=0]\,a_t^{\mathrm{WAM}}, $$

where $g_t$ is the conjunction of validity and progress gates. The WAM branch is the same-pass action prediction; the code branch denotes actions produced by executing the accepted program. This arbitration is not an expected-value comparison. A syntactically valid program with known calls and new progress can still be worse than direct action prediction, because the gate checks executability rather than whether code will produce a better trajectory.

RoboTwin: clean precision and randomized robustness trade off

The simulation evaluation uses 50 RoboTwin 2.0 tasks and 50 rollouts per task under both clean and randomized conditions. Domain randomization changes backgrounds, tabletop textures, lighting, tabletop height, and distractors. AW-0 default reaches 80.8% clean success and 20.6% randomized success, for a 50.7% average. Hybrid execution shifts that profile to 63.1% clean and 29.2% randomized, for a 46.2% average.

Method	Clean	Randomized	Average
AW-0 default	80.8	20.6	50.7
AW-0 hybrid	63.1	29.2	46.2
π0.5	54.9	27.7	41.3
WLA	64.8	3.6	34.2
DreamZero	33.6	17.6	25.6
EgoScale	26.2	2.9	14.5

The hybrid contour is revealing. It is the best AW-0 configuration under randomization, but it gives up 17.7 points in the clean setting. On successful hybrid episodes, generated code drives 14.5% of steps in clean scenes and 35.0% under randomization. This is consistent with code contributing more when visual conditions are unfamiliar, but it is not a controlled estimate of the value of runtime arbitration. The hybrid model is trained with demonstrations collected by executing code-as-policy programs, whereas the default model is trained on the standard task demonstrations. The two pipelines change both the training distribution and execution mode, so the table cannot isolate the arbitration gate alone.

The drop under randomization is also a reminder that the learned world model still sees pixels. AW-0 default falls below π0.5 in that condition, even though its code stream is visually grounded through a ViT. Code provides semantic structure; it does not make the action or video experts invariant to an unfamiliar visual domain.

Eight real-world tasks: the strongest signal is semantic skill execution

Real-world evaluation uses two robot platforms. YAM has two arms and parallel grippers. Tianji with Sharpa hands has two 22-DoF tactile dexterous hands. Both use three RGB views and a unified 64-D state-action space. Each side of that space contains a 9-D end-effector pose, 22 hand joints, and one gripper dimension. YAM leaves the hand block unused; the dexterous platform leaves the gripper slot unused. RoboTwin instead uses its native 14-D joint actions.

Each real task receives 100 demonstrations for post-training and is evaluated with three training seeds and 16 rollouts per seed, for 48 rollouts per task. Initial object poses are randomized. AW-0 averages 71.1 rubric score across the eight tasks, compared with 46.0 for DreamZero, 43.6 for π0.5, 39.9 for WLA, and 35.9 for EgoScale.

Method	Average score
AW-0	71.1
DreamZero	46.0
π0.5	43.6
WLA	39.9
EgoScale	35.9

Task	Platform	AW-0 score
Arrange Fruits	YAM, parallel grippers	91.7
Pack Backpack	YAM, parallel grippers	64.6
Replace Lightbulb	YAM, parallel grippers	75.0
Fold Towels	YAM, parallel grippers	84.4
Fruits with Tongs	Tianji plus Sharpa, dexterous hands	54.2
Deal Poker	Tianji plus Sharpa, dexterous hands	58.3
Flip Scoreboard	Tianji plus Sharpa, dexterous hands	68.8
Solve Cube	Tianji plus Sharpa, dexterous hands	72.1

Arrange Fruits and Flip Scoreboard are particularly useful evidence for the central hypothesis. They require selecting or counting the right objects rather than only reproducing a generic manipulation motion. AW-0 reaches 91.7 and 68.8 respectively. The paper also reports a separate hybrid test on the YAM Fold and Stack Towels task: 90.6 versus 84.4 for direct action-chunk execution. That comparison uses the same post-trained model and task rubric, but it covers one task and one embodiment. It does not establish that hybrid execution is broadly superior.

Ablations: code, ordering, and denoising all matter

The architectural ablations are informative because they test where code should enter. The authors compare no code, $V\rightarrow C\rightarrow A$, and $C\rightarrow V\rightarrow A$. Adding code through video improves the no-code model, but it limits code to reasoning over already predicted dynamics. The chosen $C\rightarrow V\rightarrow A$ ordering lets code shape both future-video and action representations during every denoising iteration. This is the mechanism behind the claim that planning is not merely appended to a world model.

Visual conditioning also matters. Reusing Wan2.2 VAE features for the code expert yields a 55.6 average, while a dedicated Qwen3-VL ViT reaches 71.1. The largest gap is Pack Backpack, 64.6 versus 39.6. VAE features preserve information useful for pixel reconstruction, but the code expert needs semantic distinctions such as object identity and task state. The ViT is not merely a larger encoder; it is aligned with selecting the right skill and object.

Variant	Average score	Inference latency on RTX 5090	Interpretation
Joint masked diffusion code	71.1	0.23 s	Code co-evolves with video and action
Autoregressive code	54.5	0.55 s	Serial prefix error and slower decoding
No code expert	32.9	0.19 s	Video and action alone lose explicit skill structure
VAE features for code	55.6	Not the main comparison	Reconstruction features are weaker for semantic grounding

Removing code cuts latency from 0.23 to 0.19 seconds but drops the average score from 71.1 to 32.9. Autoregressive code generation scores 54.5 and takes 0.55 seconds. Joint denoising is faster than autoregressive decoding and better on seven of eight tasks. The exceptional task is Replace Lightbulb, where autoregressive code scores 79.2 versus 75.0. That exception is useful: joint denoising is a strong design choice, not a guarantee that every task improves.

The annotation content is ablated as well. Full programs beat programs without comments on all eight shared tasks. Removing reusable skill calls lowers the average, with Deal Poker falling from 58.3 to 42.0. Natural-language subgoals are competitive on some tasks and even score 58.3 on Fruits with Tongs, but full programs beat them on seven of eight tasks. The result supports a combination of structured calls, reusable skills, and explanatory comments rather than any one label format in isolation.

What the paper has not solved

AW-0’s main limitation follows from its data pipeline. The model can only learn the skill vocabulary and contact patterns represented by its annotation library and corpus. The pretraining study freezes 50 skills across YAM and human hands. A task that needs a new abstraction, an unusual tool affordance, or a precise object relation may be labeled poorly or forced into the wrong existing skill. Better annotation throughput does not eliminate annotation bias.

A second limitation is argument correctness. Contact-event matching checks the temporal shape of a program, and the human pipeline checks hand motion, but neither validates every declaration. Wrong object attributes, counts, or coordinate arguments can pass. This matters in both modes. In default mode, bad code can misguide video and action. In hybrid mode, a valid-looking program may also execute a wrong skill call against the live environment.

Third, visual distribution shift remains severe. The simulation hybrid mode improves randomized success, but the default model loses to π0.5 under the same randomization. On hardware, camera framing and placement are identified by the deployment documentation as a major real-world failure source. A semantic plan cannot recover if the code expert sees an observation distribution outside its grounding regime.

Fourth, the hybrid arbiter is an admissibility gate, not a benefit predictor. It checks stage novelty, dependencies, syntax, allowed calls, and vacuity. It does not estimate the expected score of code execution versus the action chunk. The clean-setting loss shows why that distinction matters. A system that executes every valid program can be less reliable than one that always trusts action, even when code is capable of driving a substantial fraction of successful steps.

Finally, the real-world hybrid evaluation is narrower than the paper’s overall real-world suite: one Fold and Stack Towels task on YAM. The eight-task evaluation is for the default AW-0 policy. Deployment also remains open-loop between replanning calls and has no online learning from execution feedback. The model can replan after the next observation, but it does not update its weights or skill library from a failed attempt.

Why this is more than “VLA plus code”

The incremental value of AW-0 lies in the training contract. A conventional pipeline might ask a VLM for code, convert that code into text, and feed the text into a separate policy. AW-0 instead trains code against a frame-aligned annotation target and lets it participate in the same denoising process as future video and action. Code is both an output and a latent control signal. The video expert can use it to structure prediction; the action expert can use it to disambiguate what physical outcome the current subtask is trying to produce.

This also changes the economics of evaluation. A normal success rate tells us whether actions work. AW-0 additionally exposes what plan the model generated, what future it imagined, and how the action stream depended on both. Those outputs are not proof of interpretability, but they create concrete artifacts for debugging: a wrong skill call is different from a correct call conditioned on a wrong object, and both are different from a plausible plan whose generated video drifts off distribution.

The strongest evidence in the paper is not that code should always control the robot. Default execution performs better on clean RoboTwin and wins all eight real tasks on average. The stronger claim is that an explicit generated program can improve a world action model’s semantic reasoning when it is trained as part of the generative process. Hybrid execution then becomes a separate deployment policy with its own robustness benefit and its own failure risks.

What to watch next

The next useful experiments are fairly concrete. First, a hybrid comparison that changes only runtime arbitration, while keeping the same demonstrations and post-training budget, would separate the value of code execution from the value of code-collected training data. Second, an argument-level verifier or held-out execution check could reduce the gap between syntactic validity and semantic correctness. Third, continual updates from successful and failed executions could turn the frozen skill vocabulary into a system that learns new abstractions rather than only selecting from an initial library.

Evaluation should also test the boundary between code guidance and visual foresight more sharply: novel object instances with familiar skills, familiar objects with novel counts, ambiguous instructions requiring composition, and contact-rich tasks under camera or lighting shift. Those cases will reveal whether AW-0’s three-stream coupling is a general mechanism or an advantage concentrated in tasks that happen to match the annotation vocabulary.

AW-0 is best read as a new interface between symbolic planning and predictive control. The program is not a decorative explanation, and the future video is not just an auxiliary visualization. Both are learnable intermediate representations that shape the action chunk. The open question is whether that interface remains useful when the environment, the skill library, or the task moves beyond the distribution that produced the annotations.

Sources and resources

• AW-0 project page
• Paper: Agentic World Action Models
• Code repository
• Pretrained checkpoint on Hugging Face

Original source: https://agentic-wam.github.io/. Published research summary for RobotWorld on 2026-10-26.