Thinker-WM: A Unified World Model with Shared Multimodal Representations for Embodied Intelligence

How the Shared Representation Is Built

Visual latents and metric depth

RGB and depth are compressed by the frozen Wan2.2 VAE. At 640×480, a 49-frame window becomes a $13\times30\times40$ latent grid. Patching with kernel $(1,2,2)$ yields $13\times15\times20=3900$ tokens for each visual stream. The two modalities therefore occupy the same spatiotemporal topology, but use separate input and output projections.

This choice avoids two failure modes. Treating depth as an ordinary RGB channel ignores its physical range and noise profile. Isolating depth in an auxiliary branch prevents geometry from influencing the generated action state. Shared coordinates preserve compatibility; dedicated projections preserve semantics.

flowchart LR
    RGB["RGB 49x480x640"] --> VAE["Wan2.2 VAE"]
    DEP["Metric depth"] --> LOG["Log codec 0.15 to 4.0 m"]
    LOG --> VAE
    VAE --> LAT["Latent 13x30x40"]
    LAT --> PAT["Patch 1x2x2"]
    PAT --> VIS["3900 visual tokens"]
    ACT["Robot action"] --> SLOT["Unified slots U=20"]
    SLOT --> REL["Relative EEF and validity mask"]
    REL --> ATOK["288 steps to 48 action tokens"]
    TXT["Task Scene Action"] --> VLM["Frozen Qwen3-VL"]
    VLM --> CTX["Language context"]
    VIS --> UDT["Shared dynamics Transformer"]
    ATOK --> UDT
    CTX --> UDT
    UDT --> OUTV["RGB decoder"]
    UDT --> OUTD["Depth decoder"]
    UDT --> OUTA["Action decoder"]

The released codec uses a logarithmic mapping over a configured metric range. Its forward transform can be written as:

$$v=1-\frac{\log z-\log z_{\mathrm{near}}}{\log z_{\mathrm{far}}-\log z_{\mathrm{near}}}.$$

The example inference configuration uses 0.15 m to 4.0 m. Logarithmic spacing allocates more resolution to nearby surfaces, where manipulation usually occurs, while compressing distant depth differences. A separate decoder reverses the transform into metric depth.

A common action space

Robot datasets expose different coordinate frames, dimensions, and end effectors. Thinker-WM maps each robot into a 20-dimensional unified action space containing dual-arm end-effector displacement, relative 6D rotation, and gripper state. A semantic slot map and validity mask handle missing degrees of freedom:

$$\tilde a_u=\begin{cases}a_{\pi_r(u)},&\pi_r(u)\geq0,\\0,&\pi_r(u)=-1,\end{cases}\qquad m_u=\begin{cases}1,&\pi_r(u)\geq0,\\0,&\pi_r(u)=-1.\end{cases}$$

Invalid slots are excluded from noise injection and loss. Without this mask, zero filling would teach the model that an absent joint should remain stationary. The representation also anchors poses to the conditioning frame:

$$\Delta p_t=p_t-p_0,\qquad R_t^{\mathrm{rel}}=R_0^{\top}R_t.$$

This reduces dependence on global coordinates and camera pose. Relative rotations use the continuous 6D parameterization. Each embodiment receives eight learnable tokens, concatenated with the language condition:

$$c=[\,r_{1:8}\;;\;h_{\mathrm{und}}\,].$$

Video and action follow a fixed 6:1 temporal ratio. Forty-eight video frames correspond to 288 action steps; temporal convolution reduces these to 48 action tokens of width 3072. The action sequence thus shares the visual temporal length while remaining compact.

Figure 2: Modality-specific projections feed a shared dynamics stage; robot and language tokens provide context.

Unified Dynamics Transformer

The latent streams are concatenated and processed jointly:

$$z'=\mathcal{U}\left([\,z_v^0\,;\,z_d^0\,;\,z_a^0\,],h_{\mathrm{und}}\right).$$

During pretraining, the attention graph is deliberately directed. RGB and action read one another, while depth reads RGB, action, and itself; RGB and action do not read depth. Imperfect depth pseudo-labels can therefore shape shared weights through their loss without directly perturbing the pretrained visual and motor pathways.

Post-training opens the reverse direction when training is restricted to higher-quality robot depth. Appearance and action can then read geometry, making depth a co-evolving state rather than a one-way auxiliary target. The public implementation expresses this in `thinkerwm/uni/branch.py:100`, where modality-specific key/value sources define the directed attention graph.

Language conditioning retains both the original representation and its layer-wise update. For UDT block $k$, keys and values combine both versions:

$$K_k=W_K[\,h_{\mathrm{und}}^0;\,h_{\mathrm{und}}^k\,],\qquad V_k=W_V[\,h_{\mathrm{und}}^0;\,h_{\mathrm{und}}^k\,].$$

This preserves the original task intent while allowing the interpretation of that intent to adapt to the current multimodal state. Keeping only the original embedding would make the condition rigid; keeping only the updated one could wash out information needed later in generation.

Six functions, one parameterization

Each function mode explicitly determines whether a modality is generated, supplied as a clean condition, or omitted. A clean condition is assigned zero flow time, which gives $x_{\sigma=0}=x$ and excludes its prediction loss.

The training ratio is $0.5:0.1:0.1:0.1:0.1:0.1$. Half the steps model the full joint distribution, while every auxiliary function is exercised regularly. The mode is a deterministic function of the global step and is identical across data-parallel ranks, avoiding divergent branch sets during synchronous optimization.

Figure 3: Directed attention in pretraining. Depth reads RGB and action without feeding back into them.

Figure 5: The six function modes supported by the same network and parameter set.

Flow Matching and Staged Training

For modality $m\in\{v,d,a\}$, the clean representation is interpolated with Gaussian noise and the network regresses the corresponding velocity:

$$x_{m,\sigma_m}=(1-\sigma_m)x_m+\sigma_m\epsilon_m,\qquad u_m^\star=\epsilon_m-x_m.$$

The noise grid is shifted according to:

$$\sigma(s)=\frac{\mathrm{sh}\cdot s}{1+(\mathrm{sh}-1)\cdot s}.$$

RGB and depth use a shift of 6.0; action uses 1.0. Large visual token sequences benefit from more training mass at high noise, where global scene structure is established. The compact, smooth action sequence is better served by a nearly uniform grid. At inference, visual modalities use 50 denoising steps and action uses 10.

The action objective excludes unavailable dimensions:

$$\mathcal{L}_a=\mathbb{E}\left[\frac{\sum_{u=1}^{U}m_u\|\hat u_{a,u}-u_{a,u}^{\star}\|_2^2}{\sum_{u=1}^Um_u+\varepsilon}\right].$$

For a function mode $r$, only the generated set $\mathcal{G}(r)$ contributes to the total loss:

$$\mathcal{L}_{\mathrm{total}}=\sum_{m\in\mathcal{G}(r)}\lambda_m\mathcal{L}_m(\theta;r).$$

Training follows a staged curriculum with replay rather than a single static mixture:

$$\mathcal{D}_s=(1-\epsilon_s)\mathcal{D}_s^{\mathrm{main}}+\epsilon_s\mathcal{D}_s^{\mathrm{replay}}.$$

Stage 0 is the inherited Wan2.2 visual prior. Stage 1 adapts the video path to robot and egocentric views. Stage 2 introduces actions and geometry jointly. Stage 3 specializes the system to the target robot with reliable depth. The monotonic EWMBench progression from 2.63 to 3.34, 4.44, and 4.66 suggests that these stages contribute measurable capability rather than merely organizing the experiment.

In joint modes, video and action share a base noise level and receive bounded independent jitter:

$$\tilde\sigma_m=\mathrm{clamp}(u+\delta_m,0,1),\qquad\delta_m\sim\mathrm{Uniform}(-0.10,0.10).$$

The shared base keeps denoising progress aligned; the jitter prevents the model from seeing only exact noise-level matches. The released code uses distinct visual and action sigma grids, an implementation detail that should be respected when reproducing the checkpoint behavior.

# Condensed inference interface from the official release
# thinkerwm/thinker_wm/infer.py:355
def inference_step(..., num_inference_steps=50,
                   action_num_inference_steps=None,
                   predict_video=True, predict_action=None):
    # None or all-zero inputs denote missing modalities.
    # Conditional modalities enter as clean observations.
    ...

The interface exposes video and action prediction independently, matching the six-mode training table. The repository is an official inference release; the Data Factory, complete training corpus pipeline, and full optimization recipe are not included.

Evaluation

EWMBench

EWMBench evaluates 21 episodes from seven manipulation tasks. The model produces three candidates per episode, selects one by Hausdorff distance, and reports the sum of eight normalized dimensions over 63 videos. Thinker-WM reaches 4.66, compared with 4.60 for Qwen-RobotWorld. The larger separation appears in trajectory-sensitive measures: nDTW rises from 0.6708 to 0.7102 and dynamics from 0.3429 to 0.3630.

The comparison makes a useful distinction between appearance and embodied motion. Veo3 scores 0.8415 on scene consistency and 0.8965 on CLIP, but only 0.2130 on HSD. Thinker-WM gives up a small amount of scene score while improving trajectory and dynamics. That pattern is consistent with action participation in the generative process rather than action being a detached output.

Action-conditioned generation provides the sharper control test. Supplying the ground-truth action sequence raises Overall to 5.03, HSD to 0.7369, and nDTW to 0.8736. Dynamics changes only from 0.3630 to 0.3636. The action branch therefore affects path-sensitive measures most strongly; it does not lift every metric by a uniform amount.

WorldOlympiad and WorldArena

WorldOlympiad Robotics contains 400 manipulation episodes from more than ten physical embodiments and several camera placements. Thinker-WM scores 0.740 overall, 0.531 for 3D consistency, and 0.765 for interaction fidelity. The aggregate is the equal-weight mean of three tracks:

$$S_{\mathrm{All}}=\frac{1}{3}\left(S_{\mathrm{phys}}+S_{\mathrm{3D}}+S_{\mathrm{interact}}\right).$$

WorldArena Track-1 reports a 12-metric EWMScore. The paper excludes Dynamic Degree, Flow Score, and Motion Smoothness because of documented frame-sampling sensitivity. Thinker-WM scores 71.73, ahead of DexWorldEngine at 71.01 and BWM-Fast at 70.97. Its Depth Accuracy is 92.49 and Photometric Consistency is 46.33, but Image Quality is only 37.26, showing that the aggregate win is not a uniform lead on every dimension.

Depth and closed-loop control

On 48 AgiBot validation sequences, generated depth reaches 0.1360 AbsRel and 0.1790 m RMSE, versus 0.1625 and 0.2075 m for RynnWorld-4D. A range-normalized comparison changes the baseline AbsRel only from 0.162517 to 0.162474, so the gain is not explained by Thinker-WM's 0.15 to 4.0 m validity interval.

Adding a depth stream with one-way coupling improves success from 60.0% to 63.3%; bidirectional coupling reaches 75.0%. The result indicates that predicting geometry and using geometry during action generation are distinct capabilities. The full model also outperforms both alternatives at every evaluated checkpoint. Across 50 RoboTwin 2.0 tasks, it reports 92.6% in Clean and 91.4% in Randomized settings, a 1.2-point drop.

What the motion probe establishes

The motion probe tests whether action-relevant information is recoverable from frozen UDT video tokens. An offline teacher derives 3D point trajectories with SpatialTrackerV2 and aligns them to recorded end-effector motion using a per-window Sim(3) transform:

$$\tilde C_t=sRC_t+t,\qquad \Delta\tilde C_t=sR\Delta C_t.$$

Translation cancels after temporal differencing, leaving a scaled and rotated visual displacement target. A small head predicts two-hand displacement, and its loss combines endpoint error with magnitude error:

$$\mathcal{L}_{\mathrm{motion}}=\frac1{|\mathcal{T}|}\sum_{t\in\mathcal{T}}\|x_t-y_t\|_2+\frac1{|\mathcal{T}|}\sum_{t\in\mathcal{T}}\left|\|x_t\|_2-\|y_t\|_2\right|.$$

On 96 held-out AgiBot windows, action MSE falls to 0.116 and decreases across checkpoints. This does not prove that video tokens are actions, but it does show that executed motion is increasingly recoverable from a frozen shared representation.

Figure 6: The offline teacher and light motion head used to test the frozen video representation.

Code and Reproducibility

The official repository at commit `44cdb51` is organized as an inference release. `configs/inference_example.yaml` documents a 20-dimensional unified action space, a 6:1 video-action ratio, a logarithmic depth codec, and multiple robot mappings. The action code at `thinkerwm/thinker_wm/action_repr.py:161` reconstructs a pose from a relative 6D rotation:

# thinkerwm/thinker_wm/action_repr.py:161
def relative_rotation_6d(state_rot, action_rot, repr_name, quat_order="xyzw"):
    r_s = repr_to_matrix(state_rot, repr_name, quat_order)
    r_i = repr_to_matrix(action_rot, repr_name, quat_order)
    r_rel = r_s.transpose(-1, -2).unsqueeze(-3) @ r_i
    return matrix_to_6d(r_s), matrix_to_6d(r_rel)

This implementation corresponds directly to the paper's relative pose equation. `thinkerwm/thinker_wm/depth_codec.py:20` supplies the other clear method-to-code mapping: it converts metric depth into the latent codec domain and back with an explicit inverse.

Limitations

The authors explicitly identify the absence of real-robot closed-loop validation as the primary open problem. RoboTwin provides a meaningful simulation test, but it does not cover hardware latency, sensor noise, contact uncertainty, safety limits, or long-term recovery on a physical platform.

Depth supervision covers roughly 4,000 of the 10,000 robot hours. The one-way pretraining attention reduces the influence of noisy geometry, but most robot video still lacks reliable metric depth. The geometric representation is therefore trained on a materially narrower distribution than appearance and action.

Long-horizon prediction is implemented by chaining 48-frame inference windows at test time. The model is not trained directly on its own generated long trajectories, and the paper does not provide a long-horizon drift curve. Strong short-window metrics are not enough to establish stability after many autoregressive steps.

The WorldArena headline uses twelve metrics and omits three motion measures. The exclusion has a protocol rationale, but it also means that “best 12-metric score” should not be read as “best on every benchmark dimension.” All headline results are reported by the authors and have not been independently reproduced.

Finally, inference uses 50 visual steps and 10 action steps, yet the paper reports no deployment latency, memory footprint, or throughput. A model intended for manipulation must eventually be evaluated under the timing constraints of closed-loop control, not only under offline quality metrics.

Conclusion

Thinker-WM turns representation sharing into a concrete systems argument. A unified action interface handles embodiment differences, a shared UDT lets appearance, geometry, and action update one another, six explicit modes reuse the same parameters, and the motion probe asks whether the frozen visual state retains action-relevant structure. The reported results support this design on open-loop embodied generation and simulated closed-loop control.

The next evidence must be harder than another qualitative rollout. Real-robot closed-loop trials, long-horizon drift measurements, and deployment-cost analysis are required before benchmark leadership can be translated into dependable robot infrastructure.

Sharing is not forcing every modality into one token; it is making each modality explain the same evolving world while preserving the boundary through which it can be acted upon.