Fast-VLA: You Only Think Once for Robot Manipulation

Method overview: establish task memory, then query state continuously

Given instruction $L$ and the initial multi-camera view $I_0$, the frozen vision-language model constructs task memory:

$$M_0 = G_{\mathrm{VLM}}(L,I_0)$$

$M_0$ is not a language summary or a single pooled vector. It contains prefix keys and values from multiple model layers. The RoboTwin setup uses three initial images, each represented by 256 soft image tokens. These image tokens join the instruction tokens in the prefix. The system caches keys and values from 34 layers, preserving visual associations from early layers and task-level organization from deeper layers.

At each action-chunk boundary, the online actor receives current images $I_t$, optional history $H_t$, and robot state $q_t$. It first encodes the online evidence:

$$Z_t = G_{\mathrm{online}}(I_t,H_t,q_t)$$

The actor then changes this evidence into action queries, reads the fixed memory, and predicts the next $K$ actions in one pass:

$$\hat{A}_{t:t+K-1}=F_{\mathrm{pred}}(Z_t;M_0)$$

This separation does not remove feedback. Observations and states continually rewrite online features and queries. Only the episode-level KV memory remains fixed. The actor executes part of a chunk, receives a new observation, and predicts again. "Think once" therefore means that the vision-language model performs task contextualization once, not that the robot predicts one motion and ends the episode.

flowchart LR
  I["Initial multi-view images
and instruction"] --> V["Frozen DM0.5
one prefix pass"]
  V --> M["34-layer KV
task memory"]
  O["Current views
history and robot state"] --> D["DINOv2 encoding
and query fusion"]
  M --> X["Cross-memory attention
task retrieval"]
  D --> X
  X --> H["Action head
one forward pass"]
  H --> A["RoboTwin 50 x 14
LIBERO 10 x 7"]

The method in detail: how a fixed memory supports live control

1. Episode memory is a set of keys and values from 34 layers

The paper writes task memory as a layer-wise collection:

$$M_0=\{(K_0^\ell,V_0^\ell)\}_{\ell=1}^{34}$$

Here $K_0^\ell$ and $V_0^\ell$ are the keys and values for the instruction-plus-initial-image prefix at layer $\ell$. Early tokens tend to encode local appearance and position. Deeper tokens capture object relations and task semantics. Keeping only the final layer discards this hierarchy. The ablation confirms that all 34 layers outperform a last-layer cache on both standard LIBERO and LIBERO-Plus.

The frozen model does not need to run for every training action sample. The repository precomputes KV banks for the training episodes. The RoboTwin bank occupies 22 FP8 shards and about 1.6 TiB; the LIBERO bank occupies two BF16 shards and about 150 GiB. This makes the cost tradeoff explicit. Online repeated inference is replaced by offline cache storage and one high-cost prefill per episode.

2. Online visual encoders observe what the memory cannot

The RoboTwin actor uses a DINOv2 ViT-B/14 encoder for the head camera and two wrist cameras. Frames from the same camera share encoder weights, while learned camera-slot and time-slot embeddings distinguish their source. The history offsets are -17, -8, and 0. This gives two past head frames plus the current three-view observation. These history frames describe recent motion and occlusion; they are not written back into task memory.

The CLS token from each wrist image is combined with the corresponding seven-dimensional arm state. The final state dimension is associated with the gripper. Each arm's query receives its own wrist CLS feature and state, reducing the chance that left and right conditions are exchanged. A shared cross-visual attention stage still lets every query inspect visual tokens from all cameras.

3. Time and arm indices give action queries a stable structure

The actor initializes a query for future step $\tau$ and arm $a$:

$$z_{\tau,a}^{(0)}=e_{\mathrm{time}}(\tau)+e_{\mathrm{arm}}(a),\quad \tau\in\{0,\ldots,49\}$$

RoboTwin uses $K=50$, two arms, and seven action dimensions per arm, so one call returns a 50 x 14 absolute joint chunk. LIBERO uses one arm, $K=10$, and seven dimensions, producing a 10 x 7 chunk. Time embeddings prevent all future steps from starting with the same query. Arm embeddings give the left and right manipulators explicit roles in self-attention.

Current wrist CLS features and state are projected, concatenated with a role embedding, and passed through an MLP to form arm-local feedback. A gated residual adds this vector to the matching arm queries at every future time step. The local condition is reinjected before each of the four decoder layers, preventing depth from washing out the current arm pose and gripper state.

4. Decoder layers inspect the scene before retrieving task context

Each decoder layer applies self-attention, cross-visual attention, cross-memory attention, and an FFN. Self-attention coordinates future steps and the two arms. Cross-visual attention reads current and historical camera tokens, answering where the object and hand are now. Cross-memory attention adds the instruction context and initial-scene anchors, answering what the task requires and which objects matter.

This order has a practical interpretation. An action must first attach to the visible state, then use task memory to resolve local ambiguity. A policy that reads only fixed memory can know the destination but fail to follow a moved object. A policy that reads only the current frame can become distracted by a visually similar object. Both pathways are needed for grounded manipulation.

Figure 2: The frozen model processes only the instruction and initial views. The online actor fuses current and historical vision with robot state and time/arm queries. Four decoder layers alternate between visual evidence and task-memory retrieval before a single action head predicts the chunk.

5. The KV adapter does more than average 34 caches

Each source layer stores keys in its own rotary-position space. Averaging them directly would mix incompatible phases. Fast-VLA first removes each source layer's RoPE, projects every key into a canonical coordinate system, and then learns a weighted fusion:

$$\alpha_{m,\ell}=\operatorname{softmax}_{\ell}(s_{m,\ell})$$

$$(\bar K_m,\bar V_m)=\sum_{\ell=1}^{34}\alpha_{m,\ell}(\mathcal{R}_{c\leftarrow\ell}(K_0^\ell),V_0^\ell)$$

Here $m$ identifies the action decoder layer and $\mathcal{R}_{c\leftarrow\ell}$ maps from source RoPE coordinates into the common frame. The fused key is normalized, linearly projected, and rotated with the target layer's position encoding. The value uses a separate projection. Thus all decoder layers share one stored cache but can select different combinations of source depth.

The official implementation shows that routing is input dependent. The weight includes a learned layer bias and a router applied to the mean action query:

$$w_m=\operatorname{softmax}(\alpha_m+\operatorname{Router}_m(\bar z_m))$$

The router's final layer starts at zero, so training begins close to the intended depth-aligned cache layer. As the actor trains, live visual and state queries can alter which memory depths are most useful.

6. Actions are regressed in one forward pass

After four decoder layers, a linear action head predicts the complete chunk. The online actor, including its visual encoder, is trained with an L1 loss on normalized actions:

$$\mathcal{L}_{\mathrm{act}}=\frac{1}{KD}\sum_{k=0}^{K-1}\sum_{d=1}^{D}\left|\hat a_{t+k,d}-a_{t+k,d}\right|$$

Here $D$ is the action dimension. Unlike a multi-step generative action expert, this head has no iterative denoising, particles, or persistent noise state. It gives up explicit multimodal sampling in exchange for predictable one-pass cost. The paper attributes speed to the full system: removing the large model from the loop, using four compact decoding layers, and regressing the chunk in one pass.

7. Amortized latency includes the one-time prefill

For an episode with $H$ environment steps and chunk length $K$, let $T_{\mathrm{prefill}}$ be the vision-language prefill and $T_{\mathrm{actor}}$ the time for one actor call. The reported per-step cost is

$$\frac{T_{\mathrm{prefill}}+\lceil H/K\rceil T_{\mathrm{actor}}}{H}$$

This definition prevents the one-time model call from disappearing from the comparison. RoboTwin prefill takes 108.87 ms. Its first chunk takes 153.74 ms, later chunks take 44.87 ms, and the observed average episode length produces 1.437 ms per environment step. LIBERO prefill takes 88.38 ms, each actor call takes 34.25 ms, and amortization over 171.9 average steps gives 3.975 ms per step.

8. The repository makes the cache lifecycle explicit

In fastvla/policy.py, reset() clears the cache, the first act() call invokes build_cache(), and later calls reuse the stored keys and values for the rest of the episode. If a requested history offset is absent, the implementation falls back to the current frame. This makes the serving interface robust, but deployment code must still supply the intended history because missing frames do not raise an error.

In fastvla/model/cache_attn.py, prepare_canonical_kv() removes RoPE in FP32 before fusion is performed, reducing accumulated phase error across 34 layers. KVFusion then learns source weights, normalizes and projects keys, and applies the target layer's RoPE. In fastvla/model/fastvla.py, QueryGrid implements the time-arm query lattice, while LocalCondition combines wrist CLS features, state, and role embeddings. These files provide concrete checks on the paper's architectural description.

9. Training scale exposes the intended operating regime

RoboTwin training covers 50 tasks in Clean and Randomized settings: 27,500 episodes, 200,000 optimization steps, an effective batch of 256, and about 150 accelerator-hours. LIBERO uses 2,000 episodes across 40 tasks in four suites, 30,000 steps, batch 32, and about five accelerator-hours. Visual encoders start from DINOv2 and are fine-tuned, while the action decoder and head are trained from scratch.

These numbers support a specific claim: a compact actor can learn to read a frozen vision-language representation in simulation at moderate training cost. They do not establish zero-shot transfer to a real robot or robustness to a new sensor distribution. Prefill also peaks near 12 GiB, making the system better suited to a workstation or separate inference node than a low-power controller embedded in the robot body.

Experiments: efficiency, success, and dependence on memory

The evaluation is broad. RoboTwin runs 100 episodes per task in Clean and Randomized conditions, for 10,000 rollouts. Standard LIBERO uses four ten-task suites with 50 initial states per task, for 2,000 rollouts. LIBERO-Plus covers all 10,030 perturbation instances. Beyond success, the paper measures one action-call time, per-step amortized time, and resident memory under a shared profiling protocol.

On RoboTwin, memory reuse does not cause a large average accuracy loss. Fast-VLA reaches 92.43%, below DM0.5 at 93.45% and slightly below Faster-WAM HUST at 92.52%, but its per-step cost is 1.437 ms. DM0.5 takes 16.069 ms per step, making Fast-VLA 11.2 times faster. The corresponding gaps against Faster-WAM HUST and Fast-WAM are 10.9 and 17.1 times.

LIBERO gives a more mixed result. Fast-VLA's 95.95% is competitive on the standard benchmark, but 79.11% on LIBERO-Plus remains 6.59 points below pi-0.5 and 6.79 points below DM0.5. The authors note that both baselines use action-generating weights pretrained on external action data, while Fast-VLA trains its action decoder and head from LIBERO demonstrations. This difference does not explain every robustness gap, but it prevents a simple conclusion that memory reuse alone causes the difference.

The latency comparison must be read with the chunk-execution schedule. RoboTwin predicts 50 actions per call, which spreads fixed call overhead over more environment steps. The paper nevertheless includes the first-call prefill and uses observed episode lengths. The next question for a full robot system is whether image capture, coordinate transforms, safety limiting, and network transport erase the millisecond-level advantage. The reported numbers measure policy inference, not the complete control stack.

Attention visualizations provide evidence beyond aggregate success. In a successful 300-step block-stacking episode, head-camera attention shifts from the red block to the green block as control moves between arms. Each wrist camera tracks the object handled by its own arm, and the right wrist turns toward the green block before its sub-goal begins. This is not evidence of an explicit planner, but it shows that online visual querying changes with task phase instead of repeatedly reading the initial frame.

Figure 4: Across a 300-step bimanual stacking rollout, head attention shifts from the red block to the green block as the active arm changes. Each wrist camera follows the object relevant to its own arm, showing that the fixed task memory is queried with changing visual evidence.

Memory ablations: depth and joint image-language content matter

The authors vary both cache content and depth. Removing task memory drops LIBERO and LIBERO-Plus to 69.90% and 53.69%. Keeping only text-token KV reaches 93.20% and 73.84%. Keeping only the final layer reaches 94.70% and 73.70%. Full memory over instruction and initial images reaches 95.95% and 79.11%.

Memory swaps expose a sharper dependency. Replacing the language with another task reduces success to 35.18% and 29.66%. Keeping the same task but using the initial memory from another episode leaves only 6.26% and 8.36%. Swapping only the initial view gives 2.64% and 2.62%. Using a completely different task memory nearly collapses performance to 0.62% and 0.52%. The actor is therefore reading episode-specific scene information, not merely inferring a broad task category from the cache.

This result also reveals the risk. A blocked, overexposed, or object-misidentifying initial view enters a fixed memory that the actor repeatedly reads for the entire episode. Wrist images can correct local motion, but no mechanism reruns the vision-language model to check whether the original task interpretation was wrong. The efficiency of thinking once and the cost of a bad first observation are two sides of the same design.

Online inputs and refresh frequency

Removing wrist vision produces the largest input drop, to 65.60% and 33.49%. Removing robot state gives 94.30% and 72.41%. A wrist-only policy without the global view reaches 87.10% and 68.72%. Close wrist views are crucial during grasping, while the head view remains important for global localization. Fixed task memory cannot replace a missing live sensor.

More frequent refresh improves LIBERO-Plus but restores the large-model cost. Refreshing every chunk reaches 96.50% and 72.47%, increases vision-language calls to 17.37 times the once-per-episode schedule, and costs 12.391 ms per step. Refreshing every two or four chunks reaches 95.95% and 70.63%, or 95.95% and 68.24%. These trends agree with the paper's premise: task context changes slowly, but fixed memory loses some adaptability when the scene diverges from the initial state.

Memory producers also matter. PaliGemma reaches 95.35% and 73.15%, pi-0.5 reaches 94.90% and 71.48%, Qwen2.5-VL-7B reaches 95.35% and 75.55%, and DM0.5 reaches 95.95% and 79.11%. Cache sizes are about 9.8, 9.8, 31.8, and 79.7 MB. A larger model is not monotonically better. DM0.5's advantage appears in perturbation success as well as standard success.

Longer RoboTwin tasks do not immediately fail. Short, medium, and long step-budget groups score 94.00%, 90.27%, and 90.25%. The longest task, `put_bottles_dustbin`, requires 1700 steps and still reaches 87.00%. The evaluation does not demonstrate hours of continuous real-world operation, severe lighting drift, or an instruction change halfway through an episode.

Figure 6: Successful execution under several LIBERO-Plus perturbations. Even when the initial third-person view is corrupted, the cleaner wrist view supports local control while task memory preserves context derived from the perturbed initial observation.

Limitations and independent assessment

The authors explicitly state that task memory is built at episode initialization and remains fixed. The online actor responds to current observations, but changes in task goals, newly introduced instructions, and explicit high-level replanning remain unexplored. The authors propose event-triggered memory updates and high-level replanning as future work. This limitation concerns semantic invalidation of task memory, not merely whether the actor can track a moving object.

All reported evaluations are simulation benchmarks. The paper does not present a large real-robot study covering camera calibration error, reflective materials, communication delay, or contact dynamics. LIBERO-Plus adds useful visual and scene perturbations, but it does not fully represent real sensor noise. The memory-swap results further show that a bad initial observation can persist as a bad task context.

Memory cost remains substantial. Online resident memory is 1.22 GiB on RoboTwin and 0.86 GiB on LIBERO, but prefill peaks at 12.11 and 11.74 GiB. Moving the large model out of the repeated control path does not remove it from the robot's compute stack. The system still needs a capable workstation or an external inference service to build each episode cache.

The LIBERO-Plus gap should not be attributed only to fixed memory. Fast-VLA trains its action module from LIBERO demonstrations, while pi-0.5 and DM0.5 use action-generating pretraining. The strongest defensible conclusion is that Fast-VLA is efficient on RoboTwin and competitive on standard LIBERO, while its perturbation robustness needs further work and should not be hidden by the 95.95% standard score.

The most important engineering detail is cache lifetime. The public policy builds memory on the first call after reset(). A deployment must define exactly when to reset, how to collect each history offset, and whether to reuse memory after an error recovery. An incorrect episode boundary can make the actor read the previous task. Missing history silently falls back to current frames, so input degradation may not appear in the serving response.

Conclusion and outlook

Fast-VLA turns task contextualization into reusable multi-layer KV state and delegates repeated control to a compact actor. It does not pretend that fixed memory contains the current world state. Vision, proprioception, and history remain responsible for the changing scene. Task context stays stable at the episode level while action evidence updates continuously.

The experiments support this decomposition within the paper's simulation scope. Fast-VLA reaches 92.43% mean success on RoboTwin and 95.95% on standard LIBERO, while reducing amortized inference to 1.437 and 3.975 ms per step. The memory-swap and input ablations are particularly informative: task memory matters, but online vision remains essential. The efficiency comes from updating two kinds of information at different frequencies.

The next useful steps are clear. Event-triggered refresh could update memory when the scene changes sharply, a sub-goal completes, or confidence drops instead of calling the large model every chunk. High-level replanning could replace memory when an instruction changes. Real-robot trials should test cache construction, invalidation, and recovery under imperfect initial views, camera drift, and action-pretraining interactions.

Quote

Fast-VLA does not eliminate thought. It stops repeating the same task-level thought at every action boundary, preserving stable task semantics in memory while letting a continually queried actor track a changing world.