PAPER DEEP DIVE
mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs
mimic-video couples an Internet-scale pretrained video model with a flow-matching action decoder. Latent visual plans carry physical dynamics, while the decoder acts as an inverse dynamics model that outputs robot actions. Evaluations on SIMPLER-Bridge, LIBERO, and real-world bimanual manipulation show stronger performance with far less action data and faster convergence.
One-sentence summary
mimic-video couples an Internet-scale pretrained video model to a separate flow-matching action decoder, allowing latent visual plans to carry physical dynamics while action data learns only the inverse dynamics needed to drive the robot.
1. The Central Claim: Static Multimodal Pretraining Is Missing Dynamics
Vision-language-action models have made robot policies substantially better at following language and recognizing objects. Their usual foundation is a vision-language model trained on enormous collections of images and text. This foundation supplies semantics, compositionality, and open-world recognition, but it does not directly encode what happens next when a gripper closes, an object slides, or two arms exchange an object.
In the standard VLA recipe, the missing temporal and physical knowledge must be recovered from robot trajectories. That creates a data problem with two coupled costs: each trajectory is expensive to collect, and the same backbone must learn both high-level scene structure and low-level control from those trajectories. Scaling the policy therefore means scaling teleoperation, which is difficult across embodiments and long-horizon tasks.
The authors start from a different premise. Video pretraining exposes a model to temporal sequences in which objects move, hands interact, and scenes change under physical constraints. It is still not robot action data, but it is a much closer proxy for manipulation dynamics than a collection of static image-text pairs. If those dynamics can be represented well, the action-learning stage can focus on the inverse problem rather than learning the future from scratch.
Their Video-Action Model, or VAM, operationalizes this premise by pairing a generative video backbone with an independent action decoder. The video model produces an internal latent plan, and the decoder maps that plan plus proprioception into an action chunk. The architecture intentionally stops short of requiring a complete, pixel-clean future video at deployment time.

Figure 1: VLAs learn dynamics during expensive robot post-training, while VAMs acquire semantics and visual dynamics from video and use a smaller amount of action data for control.
2. A Policy Is Decomposed into a Future Planner and an Inverse Dynamics Decoder
The formal policy predicts an action chunk from a short observation history, language instruction, and robot state:
$$\pi(\mathbf A_t\mid\mathbf o_t,l),\qquad \mathbf o_t=[\mathbf I_{t-H_o+1},\ldots,\mathbf I_t,l,\mathbf q_t],\qquad \mathbf A_t=[\mathbf a_t,\ldots,\mathbf a_{t+H_a-1}].$$
One could model future video and future actions jointly, but doing so spends computation on details that may not matter for control. The paper instead exposes a compact handoff. The video backbone receives clean past latents, noisy future latents, language, and a video flow time τv. An intermediate hidden state is extracted at layer k:
$$\mathbf h^{\tau_v}=v_\phi^{(k)}(\mathbf z_{\mathrm{past}}^0,\mathbf z_{\mathrm{future}}^{\tau_v},l,\tau_v).$$
This state is not necessarily a finished visual prediction. It is an internal representation of what transformation might take the scene from its current state toward the instructed future. The action decoder then cross-attends to that representation and estimates a vector field over the action chunk.
Calling the decoder an inverse dynamics model is useful precisely because it describes the remaining task. The video representation carries the proposed scene evolution; the action decoder determines which robot controls are consistent with that evolution given the current proprioceptive state. The split lets the video system specialize in dynamics and the smaller decoder specialize in embodiment-specific control.
This is also why the authors can freeze the video backbone during action-decoder training. The action loss does not have to teach the video model how to predict the future. It only shapes the decoder's ability to consume the frozen visual evidence.
3. Conditional Flow Matching Supplies the Shared Generative Language
Both the video backbone and the action decoder are trained with conditional flow matching. A clean sample x0 is connected to Gaussian noise ε along a straight conditional path:
$$\mathbf x^{\tau}=(1-\tau)\mathbf x^0+\tau\boldsymbol\varepsilon,\qquad \tau\in[0,1].$$
The conditional velocity is simply the direction from data to noise:
$$\mathbf u_\tau(\mathbf x^\tau\mid\mathbf x^0)=\boldsymbol\varepsilon-\mathbf x^0.$$
A neural vector field is trained to regress this target:
$$\mathcal L_{\mathrm{CFM}}=\mathbb E_{\mathcal T(\tau),p_0(\mathbf x^0),p_\tau(\mathbf x^\tau\mid\mathbf x^0)}\left\lVert v_\theta(\mathbf x^\tau,\tau)-u_\tau(\mathbf x^\tau\mid\mathbf x^0)\right\rVert^2.$$
Generation integrates the learned field from noise back toward data:
$$\hat{\mathbf x}^0=\boldsymbol\varepsilon+\int_1^0v_\theta(\hat{\mathbf x}^\tau,\tau)\,d\tau.$$
The continuous time variable is central to the paper. The video backbone can be stopped at an intermediate time, so the action decoder can consume a partially denoised plan. The action decoder itself integrates all the way to clean actions. These two processes need not share a flow time, and the paper explicitly trains them with independent time distributions.
4. The Video Backbone Provides a Temporally Grounded Latent Plan
The implementation uses Cosmos-Predict2, a 2B latent diffusion transformer, with a pretrained 3D tokenizer and a T5 language encoder. The clean context consists of five past frames. Future frames are represented by noisy latent patches. Transformer layers alternate temporal self-attention over the whole sequence, language cross-attention, and an MLP.
Training has two stages. First, the video model is adapted to robot video with LoRA. This changes the low-rank parameters that align the general video model with the visual and dynamic domain of the robot data while preserving most of the pretrained backbone. Second, the video model is frozen and the action decoder is trained from scratch.
The official pipeline makes the information boundary explicit. It calls the video generator with a requested stopping step and hidden layer, flattens the returned hidden states, and sends them with the corresponding video sigma into the action component:
crossattn_emb, video_sigma = self.video2world_pipeline.generate_video(
vid_input=input_vid,
is_video_embedding=False,
num_latent_conditional_frames=1 if T == 1 else 2,
prompt=prompt,
prompt_embedding=prompt_embedding,
guidance=0.0,
num_sampling_step=num_sampling_step,
return_context_at_step=stop_after_step,
hidden_state_layer_idx=self.world2action_pipeline.config.xattn_layer_idx,
)
shape = crossattn_emb.shape
crossattn_emb = crossattn_emb.reshape(shape[0], -1, shape[-1])
return self.world2action_pipeline(
state_B_HO_O=state_B_HO_O,
crossattn_emb=crossattn_emb,
context_timesteps_B_1=video_sigma.unsqueeze(1),
)
The paper describes the extraction layer abstractly as k. The released experiment configuration sets the cross-attention layer index to 20, while the implementation's index convention is zero-based. That detail matters when reproducing the code: the conceptual k in the equation should not be confused with a one-based layer count.
Passing sigma alongside the hidden state is also important. The decoder must know how noisy the visual condition is, because training exposes it to many different video flow times. A representation detached from its sigma would erase a variable that the decoder was explicitly trained to condition on.
5. The Action Decoder Mixes State Tokens, Action Tokens, and Visual Evidence
The decoder is a smaller diffusion transformer. Proprioceptive state and noisy actions pass through separate MLPs. Their token sequences are concatenated, given absolute time position embeddings, and processed by blocks that combine video cross-attention, action self-attention, and an MLP. Each block uses residual paths and AdaLN modulation.
The AdaLN conditioning encodes both τv and τa. This joint time conditioning lets the model distinguish a clean visual plan from a noisy one while also representing the current stage of action denoising. The two modalities therefore interact not only through cross-attention but also through the modulation of every decoder block.
During decoder training, the state token can be replaced by a learned mask token. The released action-model call uses an observation dropout value of 0.2. The paper describes the same idea more generally as random replacement of the proprioceptive encoding. This regularization pushes the decoder not to rely entirely on a low-dimensional state vector that may be noisy or incomplete at deployment.
obs_emb = self.obs_embedder(state_B_HO_O)
action_emb = self.action_embedder(xt_B_HA_A)
obs_dropout_mask = torch.bernoulli(
torch.full((obs_emb.shape[0], 1, 1), obs_dropout,
device=obs_emb.device, dtype=obs_emb.dtype)
)
obs_emb = (1.0 - obs_dropout_mask) * obs_emb
+ obs_dropout_mask * self.obs_mask_token
sequence = torch.cat((obs_emb, action_emb), dim=1) + self.pos_embedding
Clean actions are recovered by integrating the learned action field from noise:
$$\mathbf A_t^0=\mathbf A_t^1+\int_1^0\pi_\theta(\mathbf A_t^{\tau_a},\mathbf q_t,\mathbf h_t^{\tau_v},\tau_a,\tau_v)\,d\tau_a.$$
The action-time distribution is deliberately nonuniform:
$$\mathcal T_a(\tau_a)\propto\sqrt{\tau_a-0.001}.$$
Video time follows a logit-normal distribution inherited from the video pretraining regime. Independent video and action times force the decoder to handle conditions whose noise level does not match the current control noise level. That coupling is part of the generalization mechanism, not an implementation accident.
At inference, τv=1 has a special property. The future latent remains pure noise and no intermediate video ODE step is needed before entering the backbone. One video forward pass produces the hidden-state condition, while the action tensor still undergoes the full flow integration. The default is therefore not “no video planning”; it is a one-pass latent plan under maximum noise.
6. Closed-Loop Inference Uses Plans as Conditions, Not as Videos to Replay
flowchart LR
O[Five-frame observation history] --> Z[3D tokenizer latents]
L[Language instruction] --> E[T5 language embedding]
N[Noisy future latent] --> V[Cosmos video transformer]
Z --> V
E --> V
V --> H[Selected hidden layer plus sigma]
H --> A[Flow-matching action decoder]
S[Proprioception and action tokens] --> A
A --> C[Action chunk]
C --> X[Execute initial actions]
X --> O
The real system does not generate a complete video and blindly execute a trajectory. It maintains an observation history, calls the VAM at chunk boundaries, executes a prefix of the returned actions, and then closes the loop again. This is visible in the official SIMPLER inference wrapper, which keeps image and low-dimensional histories, constructs a new condition from the latest context, and refills an action buffer.
The design tolerates the fact that the generated visual condition is imperfect. Control is continually refreshed from new observations. Even if the latent plan is noisy or biased, execution does not commit to an unrolled video for the entire task.
Inference remains expensive relative to a compact policy. A 2B video transformer is involved even in the fastest default mode. The paper's efficiency claim concerns action-data requirements and convergence, not a claim that VAMs have lower compute than every VLA or specialized imitation policy.
7. Oracle Futures Reveal That the Decoder Is Not the Main Bottleneck
The paper's case study fixes the action decoder and changes only the source of video conditioning. It compares predicted futures from pretrained and robot-finetuned backbones with latent representations of ground-truth future video. With expert futures, decoder success approaches the ceiling even when the underlying video model was not finetuned on the target robot distribution.

Figure 2: Expert future latents make the same action decoder nearly perfect. Predicted conditions expose the much larger gap between a general and robot-finetuned video backbone.
The controlled result supports a narrow but important claim. The decoder is expressive enough to recover actions when the future representation is informative. With predicted futures, performance depends strongly on whether the video backbone was adapted to the robot data. The bottleneck is therefore the quality and distribution of the latent plan rather than an inability of the small action head to solve inverse dynamics.
The experiment does not imply that a single future frame determines all robot behavior. Contact, compliance, and hidden state can matter. Expert video acts as a strong oracle precisely because it contains the relevant visual evolution. The result says that if such a plan is available in the backbone representation, the control decoder can exploit it.
It also exposes a deployment risk. The decoder is trained on latents extracted from real future video, but at test time it receives latents shaped by the video model's own generations. Any mismatch between those distributions becomes part of the policy's effective error. The later study of video flow time can be read as a search for a condition that is both informative and robust to this gap.
8. The Most Interesting Result: Better-Looking Video Is Not Necessarily Better Control
If future video is a plan, one might expect full denoising to maximize success. The SIMPLER sweep shows the opposite for autonomous generation. Success generally rises toward high noise, and the best average is often near τv=1, where only one video forward pass is needed.

Figure 6: Each curve is one SIMPLER task. The best operating point differs by task, but most peaks lie in a high-noise region rather than at a fully denoised future.
Two mechanisms explain this behavior. First, a fully denoised generated future may be inaccurate in ways that look plausible. Even when it is accurate, it can be out of distribution relative to the real future latents used to train the action decoder. Leaving noise in the plan acts like a train-test augmentation and discourages the decoder from depending on artifacts that only appear in clean generated video.
Second, the final hidden states of a flow model may be less informative for downstream reasoning. Near zero noise, the network input is already close to the target, and the optimal transformation can resemble a small identity correction. Earlier flow times force the model to represent how the scene should move from a noisy state toward the goal. That representation can be more useful to an action decoder.
The authors separate these effects with a second oracle sweep using noisy expert futures. Action reconstruction error is lowest around τv≈0.4 and increases toward both clean and pure-noise ends. The expert case identifies an information/robustness optimum. The autonomous case moves toward higher noise because generated clean videos add their own distribution shift.
This is a stronger scientific finding than a single benchmark win. It says a video world model can be useful to control even when the decoded video is not useful as a visual artifact. The representation of transformation can matter more than pixel fidelity.
τv remains an inference hyperparameter. Per-task tuning improves SIMPLER performance, but the paper does not learn a predictor for it or establish one value across embodiments. The fast default is a reasonable engineering point, not a universal law.
9. SIMPLER-Bridge: Controlled Evidence for the Video Prior
mimic-video reaches 46.9% average success on the four SIMPLER-Bridge tasks without per-task video-time tuning. That is above ThinkAct at 43.8%, FLOWER at 45.0%, OpenVLA at 14.6%, and Octo at 16.0%. The most informative comparison is the authors' π0.5-style VLA, which uses a PaliGemma backbone and an equivalent action decoder with the same action-data regime. It reaches 35.4%.
| Method | Training regime | Average success | What it establishes |
|---|---|---|---|
| OpenVLA | Finetuned | 14.6% | Strong semantic prior alone does not solve low-level dynamics |
| Octo | Finetuned | 16.0% | Generalist policy reference |
| ThinkAct | Pretrained | 43.8% | Strong planning-related baseline |
| FLOWER | Finetuned | 45.0% | Strong flow-policy baseline |
| π0.5-style VLA | Target action decoder from scratch | 35.4% | Closest controlled comparison |
| mimic-video | Target action decoder from scratch | 46.9% | Default video flow time |
| mimic-video, per-task τv | Target action decoder from scratch | 56.3% | Inference-time specialization |
Because the action decoder and data protocol are matched, the comparison isolates the conditioning backbone more effectively than a table of unrelated published systems. The margin is not enormous, but it points in the direction predicted by the paper: temporal video pretraining provides a stronger control prior than static image-text pretraining.
Per-task τv raises the average by 9.4 points, with Put Carrot improving from 37.5% to 54.2% and Stack Blocks from 12.5% to 29.2%. Eggplant is already perfect at the default. The gain is real but task-dependent, and tuning requires evaluation rollouts that may be expensive or unsafe on a new robot.
The result should not be inflated into a universal victory. FLOWER is close in average performance, and no single benchmark covers all manipulation regimes. The paper's strongest evidence is the controlled VLA comparison, not a claim that every VAM is superior to every alternative policy family.
10. LIBERO: Competitive Accuracy with Much Less Action Data
On LIBERO, mimic-video reaches 93.9% average success: 94.2% on Spatial, 96.8% on Object, and 90.6% on Goal. It trails OpenVLA-OFT's 96.9% average, but exceeds OpenVLA at 84.1%, DiT Policy at 88.6%, and the matched π0.5-style VLA at 85.9%.
| Method | Spatial | Object | Goal | Average |
|---|---|---|---|---|
| Diffusion Policy | 78.3% | 92.5% | 68.3% | 79.7% |
| Octo | 78.9% | 85.7% | 84.6% | 83.1% |
| DiT Policy | 84.2% | 96.3% | 85.4% | 88.6% |
| OpenVLA | 84.7% | 88.4% | 79.2% | 84.1% |
| OpenVLA-OFT | 96.2% | 98.3% | 96.2% | 96.9% |
| π0.5-style VLA | 79.2% | 94.0% | 84.4% | 85.9% |
| mimic-video | 94.2% | 96.8% | 90.6% | 93.9% |
The data-efficiency study is more distinctive. The decoder reaches the maximum success previously achieved by the VLM-conditioned decoder using only 10% of the action data. Reducing the data to one episode per task, a 98% reduction, still yields 77% average success. The latter result is competitive with the reported Diffusion Policy average despite training the action decoder on only 2% of the demonstrations.
This is an action-data result, not a total-cost result. The shared video backbone is adapted on roughly 200 hours of robot video. Those hours and the video compute are a significant investment. The economic case for a VAM depends on amortizing that shared cost across many downstream tasks rather than treating it as free pretraining.
The authors also report about a 2x convergence advantage. Figure 5 shows mimic-video rising faster and approaching a higher plateau than the π0.5-style decoder under matched batch size and task-optimal learning rates. The evidence supports the reported factor as an empirical summary; deriving a new exact ratio from arbitrary curve points would overstate the figure.

Figure 5: The action decoder conditioned on video representations converges earlier and to a higher success rate, despite the VLA baseline also seeing task-specific action data during FAST pretraining.
11. Real Bimanual Manipulation: Compensating for Missing Wrist Views
The real-world platform has two Franka arms and two 16-DoF anthropomorphic hands. Its full sensing suite includes a workspace camera, four wrist cameras, and proprioception. The evaluation uses two long-horizon tasks: Package Sorting and Measuring Tape Stowing. Their action datasets contain 512 episodes over 1 hour 33 minutes and 480 episodes over 2 hours 14 minutes, respectively.
The DiT-Block baseline with only the workspace view reaches 11.0% on Packing and 30.0% on Handover. Adding wrist cameras raises those values to 42.6% and 74.1%, exposing how strongly hand occlusion affects a conventional policy. mimic-video uses only the workspace view at deployment and reaches 72.0% and 93.0%.
| Method | Packing | Handover | Deployment views |
|---|---|---|---|
| DiT-Block | 11.0% | 30.0% | Workspace only |
| DiT-Block with wrist cameras | 42.6% | 74.1% | Workspace plus four wrists |
| mimic-video | 72.0% | 93.0% | Workspace only |
The proposed explanation is that predicted dynamics bridges part of the visual uncertainty caused by occlusion. A global view may not reveal the exact finger-object contact, but the video representation can encode the expected future motion of the hands and objects. The action decoder can exploit that temporal context even when a conventional single-view policy cannot infer the hidden contact state.

Figure 4: Package Sorting and Tape Stowing. White borders show autonomous robot execution; blue borders show generated visual plans that condition control but are not replayed as video.
This is encouraging evidence for high-dimensional bimanual manipulation, but it is not a broad real-world validation. The study covers two tasks, one platform, and one deployment camera. It does not establish zero-shot transfer across embodiments, long-term operation, or new tasks without action finetuning. The authors also do not train one unified decoder across the evaluated robot families.
12. What the Released Code Clarifies
The code confirms that the video and action pipelines are separate modules. `Video2World2ActionPipeline` extracts and reshapes the selected video hidden state before invoking `World2ActionPipeline`. This modular boundary makes it possible to vary the video stopping step, cross-attention layer, and action chunk configuration independently.
It also confirms that partial denoising happens inside the video sampling path through `return_context_at_step`. The corresponding sigma is returned and passed into the action AdaLN. The decoder therefore knows not only which visual features it sees but also how noisy that condition is.
The action model uses a 0.2 state-dropout setting in the released training call. The mask replaces the state token with a learned mask token. The paper's more general wording does not name this number, so reproduction should distinguish the conceptual regularization from the implementation-specific value.
Finally, the SIMPLER wrapper rebuilds image and low-dimensional histories for each controller step and refreshes the action buffer. The system is closed-loop. Any analysis that assumes one generated video is decoded once and replayed for an entire episode misreads the deployed control path.
13. Strengths of the Decomposition
The first strength is abstraction. The video model can be shared across tasks while the action decoder absorbs embodiment-specific action scale and control frequency. This is a cleaner division than forcing every video and proprioceptive detail into one monolithic policy.
The second is controllability of inference. τv offers a continuous spectrum between a one-pass noisy plan and a more heavily denoised future. The fastest setting is also competitive, so the method does not require full video generation to obtain useful control information.
The third is diagnostic clarity. The oracle-future experiment separates representation quality from action-head capacity. Future research can improve video pretraining, layer selection, or distribution matching and ask which part of the error moves, instead of relying only on end-to-end success rate.
The fourth is a useful conceptual correction to world-model discourse. A model can be valuable to a robot even when its decoded future is not visually perfect. The relevant product is not necessarily a video that a human would judge as realistic; it can be a latent transformation that supports action inference.
14. Limitations and Open Risks
Real-world evidence is limited to two tasks and one bimanual robot. It does not test mobile manipulation, deformable objects at scale, transparent surfaces, external disturbances, or many embodiments. The single workspace view is a demanding condition, but one platform cannot establish general occlusion robustness.
Inference cost remains substantial. A 2B video transformer is still in the control loop even when only one pass is used. The paper does not provide a hardware-normalized latency table, quantify control jitter across action chunk lengths, or compare deployment memory against compact imitation policies.
Action data remains necessary. The reported 10x sample efficiency is relative to action-decoder training and does not eliminate real trajectories. Cross-embodiment use would likely require target-platform action data and could also require target-domain video adaptation.
Several design choices are empirical rather than learned: the intermediate layer, video flow-time range and default, action-time distribution, dropout, and action chunk size. Per-task τv tuning is effective but costly. The paper does not present a self-calibration procedure that can choose this parameter safely on a new robot.
The oracle study makes video representation quality the dominant scaling axis. This is a strength for analysis and a risk for deployment. If the video model hallucinates a different plan or is poorly aligned with the target embodiment, the action decoder may confidently infer actions for the wrong future.
Finally, the work does not demonstrate a single large cross-embodiment action decoder. Shared video pretraining is reusable, but the evaluated action heads are task- and platform-specific. The broader promise of a universal VAM remains an extrapolation from the current evidence.
15. Assessment
mimic-video makes a coherent case for moving part of robot-policy learning into video pretraining. The policy is not merely a VLM with a different input modality. Its architecture separates future visual dynamics from inverse dynamics, trains the two with independent flow schedules, and uses partial denoising to avoid paying for a full future video.
The benchmark evidence is strongest where the comparison is controlled: video-conditioned and VLM-conditioned decoders use the same control architecture and action-data protocol, and the video version learns much faster. The real bimanual results add a concrete demonstration of high-dimensional control with occlusion, while the τv study offers a mechanistic insight that is likely to outlast any one benchmark score.
The correct conclusion is directional rather than universal. Video representations appear to be a better substrate for action decoding than static image-text representations in the tested settings, and complete video reconstruction is unnecessary. Questions of deployment cost, cross-embodiment training, target-domain video adaptation, and reliable plan selection remain open.
“The future plan does not have to be a finished video; it only has to contain the dynamics from which the robot can infer the next action.”
Paper: arXiv:2512.15692v2. Code: https://github.com/mimic-video/mimic-video.
16. Evidence at a Glance
| Evidence | Reported result | Supported conclusion | Not supported |
|---|---|---|---|
| Expert-future oracle | Near-ceiling success | Decoder can recover actions from informative futures | Generated futures are equally accurate |
| SIMPLER-Bridge | 46.9% default; 56.3% tuned τv | Video condition beats the matched VLA condition | A universal win over every policy family |
| LIBERO | 93.9% average | Competitive multitask manipulation | Higher than all possible baselines |
| Action-data scaling | 10% reaches VLA peak; 2% yields 77% | High action-decoder sample efficiency | Shared video training is cheap |
| Real bimanual tasks | 72.0% and 93.0% | Strong single-view control under occlusion | Proven cross-embodiment generality |



