Skip to content
←Back to Papers

PAPER DEEP DIVE

VLARobot ManipulationEmbodied AI

HALO: A Unified Vision-Language-Action Model for Embodied Multimodal Chain-of-Thought Reasoning

Vision-Language-Action (VLA) models have shown strong performance in robotic manipulation, but often struggle in long-horizon or out-of-distribution scenarios due to the lack of explicit mechanisms for multimodal reasoning and anticipating how the world will evolve under action. Recent works introduce textual chain-of-thought or visual subgoal prediction within VLA models to reason, but still fail to offer a unified human-like reasoning framework for joint textual reasoning, visual foresight, and action prediction. To this end, we propose HALO, a unified VLA model that enables embodied multimodal chain-of-thought (EM-CoT) reasoning through a sequential process of textual task reasoning, visual subgoal prediction for fine-grained guidance, and EM-CoT-augmented action prediction. We instantiate HALO with a Mixture-of-Transformers (MoT) architecture that decouples semantic reasoning, visual foresight, and action prediction into specialized experts while allowing seamless cross-expert collaboration. To enable HALO learning at scale, we introduce an automated pipeline to synthesize EM-CoT training data along with a carefully crafted training recipe. Extensive experiments demonstrate that: (1) HALO achieves superior performance in both simulated and real-world environments, surpassing baseline policy pi_0 by 34.1% on RoboTwin benchmark; (2) all proposed components of the training recipe and EM-CoT design help improve task success rate; and (3) HALO exhibits strong generalization capabilities under aggressive unseen environmental randomization with our proposed EM-CoT reasoning.

Authors: Quanxin Shou, Fangqi Zhu, Shawn Chen, Puxin Yan, Zhengyang Yan, Yikun Miao, Xiaoyi Pang, Zicong Hong, Ruikai Shi, Hao Huang, Jie Zhang, Song Guo
Affiliation: The Hong Kong University of Science and Technology
Paper: arXiv:2602.21157v2
Code: qshou-coder/HALO, Apache-2.0, with training, fine-tuning, and data documentation

The short version

HALO turns robot control into a three-stage multimodal reasoning sequence: write a short plan, imagine the next task-relevant visual state, then generate an action chunk conditioned on both intermediate results. The model keeps three specialized transformer experts but couples them through shared attention. On RoboTwin 2.0 it reaches an average success rate of 80.5% in the clean setting, 34.1 percentage points above π0.

HALO architecture
Figure 1: HALO routes textual reasoning, visual subgoal generation, and action prediction through specialized experts connected by shared self-attention.

Why a direct observation-to-action policy is not enough

A standard vision-language-action model treats manipulation as a learned mapping from instruction and recent images to motor commands. That formulation is powerful when the training and deployment distributions are close. It is also compact enough for real-time control. The weakness appears when a task spans many stages or when an object, background, or lighting condition moves away from the demonstrated cases. A reactive policy can still know what a cup looks like while lacking a stable representation of which subgoal is currently active, how the scene should change after a grasp, or whether an apparent free-space path is actually compatible with the next placement.

Long-horizon manipulation makes this internal ambiguity visible. Consider a sequence that requires picking up a tool, moving it around an obstacle, contacting a target, and returning the arm to a neutral posture. A single action-chunk policy may reproduce the training trajectory when the scene is familiar, but a small visual perturbation can shift attention to the wrong object or cause the arm to terminate a motion too early. The missing ingredient is not necessarily more parameters. It is an intermediate state that preserves task structure and exposes a spatial prediction against which the action can be conditioned.

Textual chain-of-thought offers one answer. The model first decomposes the instruction into language, giving itself a structured statement of what should happen next. This can improve task sequencing and produce an explanation that a human supervisor can inspect. Language alone, however, does not fully specify the geometry of a manipulation step. A sentence such as “place the blue block on the red block” still leaves open the approach direction, the gripper orientation, the visible target location, and the expected state after contact.

Visual subgoal prediction supplies the complementary information. Instead of describing the future only in words, the policy generates a plausible image of the scene at the end of the current subtask. Such a target can encode object layout and spatial relations that are awkward to express in language. Yet an image-only reasoning path has its own failure mode. It may generate a convincing future frame without a clear account of why that frame is the correct next stage, or it may transfer appearance details while losing the symbolic identity of the object to be manipulated.

HALO treats these modalities as a sequence rather than competing alternatives. Language supplies semantic decomposition and an explicit task rationale. The visual target conditions that reasoning on the current scene and makes the intended future state concrete. The action expert then sees the instruction, history, plan, and predicted goal before producing continuous control. This ordering is the paper's central methodological claim: reasoning becomes useful for control when each intermediate representation constrains the next, and when the final controller is trained to consume all of them.

There is also an architecture argument behind the method. Images, text, and actions are not naturally represented by the same output distribution. Text is discrete and autoregressive, images are high-dimensional continuous latent fields, and actions are low-dimensional continuous trajectories with tight temporal structure. A monolithic model can support all three, but the optimization signals may interfere. HALO follows the mixture-of-transformers idea: preserve separate parameter sets for the three generative roles, then exchange information through a shared attention mechanism. The design aims to retain modality-specific inductive biases while still allowing one reasoning episode to span the full chain.

What the paper is trying to establish

The empirical argument has three parts. First, combining textual reasoning and visual foresight should outperform either modality alone. Second, multi-task pre-training should provide a foundation that is measurably necessary for downstream manipulation, not merely a generic initialization trick. Third, the resulting controller should remain useful under distribution shift in simulation and on a real bimanual robot. The experiments are organized around these claims, with one ablation removing reasoning components and another removing categories of pre-training data.

Formulating deliberation as a conditional sequence

The paper models a trajectory as $\tau=\{(\mathbf{o}_{t},\mathbf{l},\mathbf{a}_{t})\}_{t=1}^{T}$, where $\mathbf{o}_{t}$ is a visual observation, $\mathbf{l}$ is a language instruction, and $\mathbf{a}_{t}$ is a continuous action. A conventional policy with an action chunk of length $m$ learns $\pi_{\theta}(\mathbf{a}_{t:t+m}\mid\mathbf{l},\mathbf{o}_{t-k:t})$, using the previous $k$ frames as history. The policy can be expressive, but every intermediate decision is implicit.

HALO decomposes the same problem into textual reasoning, visual foresight, and action prediction. The first distribution produces a textual reason trace:

$$\mathbf{r} \sim P_\theta(\cdot \mid \mathbf{l}, \mathbf{o}_{t-k:t})$$

The second distribution predicts a visual state at a future stage boundary, conditioned on the instruction, observed history, and generated reasoning:

$$\hat{\mathbf{o}}_{t+h} \sim P_\theta(\cdot \mid \mathbf{l}, \mathbf{o}_{t-k:t}, \mathbf{r})$$

The final distribution produces the action chunk from all four sources of context:

$$\mathbf{a}_{t:t+m} \sim \pi_\theta(\cdot \mid \mathbf{l}, \mathbf{o}_{t-k:t}, \mathbf{r}, \hat{\mathbf{o}}_{t+h})$$

The notation is simple, but the causal interpretation matters. The future image is not an independent side prediction. It is conditioned on the textual plan, and the action is conditioned on both. During training, the model receives aligned supervision for each stage. During inference, the sequence can be run in full or selectively shortened, which becomes an important tradeoff for latency.

The horizon $t+h$ is tied to a subtask boundary rather than an arbitrary frame. This is a deliberate choice about what the visual target should mean. Predicting one frame far into the future would ask the model to solve multiple stages at once. Predicting every intermediate frame would create highly redundant targets and make the learning problem dominated by tiny appearance changes. A subtask terminal frame is a middle ground: it is visually specific, but it represents a meaningful completion state that the action can use as a local objective.

Inside the mixture of experts

Three specialists with separate outputs and a shared context

HALO instantiates three experts from Qwen2.5-1.5B: a multimodal understanding expert, a visual generation expert, and an action prediction expert. Their parameters are independent, giving a cumulative model size of roughly 4.5 billion parameters. They interact through shared self-attention, so a hidden state from one expert can draw on the context established by another. The separation lets the language path remain autoregressive, the visual path operate on latent image tokens with a diffusion or flow-matching process, and the action path preserve continuous low-dimensional control.

This is a practical response to a real conflict in unified models. If a single output head must represent words, image latents, and actions, the training objective has to balance completely different scales and likelihoods. Visual tokenization can also constrain the language model's semantic behavior when both capabilities are forced into the same discrete vocabulary. HALO does not eliminate the shared representation; it limits sharing to an attention context while leaving the modality-specific generation process intact.

The architecture is not three models connected by a fixed handoff. The shared attention creates a cross-expert workspace. Text tokens can influence visual generation, visual tokens can condition action denoising, and action tokens can attend to the plan and the predicted scene. The phrase “unified model” is therefore meaningful: the experts are separate computational modules, but the task is represented as one sequence with one dependency structure.

Special tokens make the phase boundaries explicit

HALO expands the Qwen2.5 vocabulary with control tokens for subtasks, plans, movements, visual generation, and action generation. A normal next-token prediction continues the textual plan. When a visual start token appears, the hidden state is routed toward the visual expert. An action start token triggers the control expert. The sequence format is therefore both a data schema and a routing mechanism.

These markers do more than select a module. They encode the intended causal order. The visual target should be generated after the textual plan, and the action should be generated after the visual target and plan exist. If the model were allowed to produce all three in arbitrary order, the training signal would no longer match the cognitive story the paper is testing. The control-token sequence makes the ordered decomposition part of the model contract.

The inference code follows the same order directly. Text generation appends the plan to the context, image generation consumes that context and adds either a decoded image or its latent representation, and action generation runs last. This code-level correspondence is useful evidence that the architecture is not merely a high-level diagram.

if use_subtask:
    generated_text = self.gen_text(gen_context, ...)
    gen_context = self.update_context_text(generated_text, gen_context)
if use_goal_image:
    goal_output = self.gen_image(..., gen_context, ...)
    gen_context = self.update_context_with_latents(...)
generated_action = self.gen_action(action_shape, gen_context, ...)

Source correspondence: the generation loop in inference/halo_inferencer.py.

The attention mask is part of the method

A shared sequence containing text, image latents, noisy generation tokens, and actions needs a carefully designed mask. Text must be causal so that autoregressive training does not reveal future words. Visual tokens within the same frame should be bidirectional because a complete image is a joint spatial object; restricting each pixel or patch to only earlier patches would make generation unnecessarily difficult. Interactions across frames and modalities must remain causal so that the visual target cannot leak future state information into the plan that is supposed to predict it.

Noise tokens introduce another leakage risk. The visual generator is trained by adding noise to a target latent. If a noisy token can directly attend to the clean target, the denoising objective becomes trivial. HALO therefore blocks noisy tokens from attending to their corresponding supervision. It also blocks non-noise tokens from attending to noisy tokens, preventing corrupted latent content from contaminating the language or action context. The resulting mask has a structured block pattern rather than a single causal triangle.

HALO attention mask
Figure 3: The attention mask separates text, semantic vision, generated vision, noisy vision, and action tokens while preventing ground-truth and noise leakage.

This detail is important when evaluating the claim that HALO supports “unified” generation. The model is not simply concatenating modality tokens and applying a normal decoder mask. The mask implements the dependency graph of EM-CoT: plans precede goals, goals precede action conditioning, and noisy latent variables remain isolated from the clean targets they are supposed to recover.

Different encoders for understanding and imagination

The understanding path uses a SigLIP2-so400m/14 vision transformer initialized from a 384×384 model. The implementation interpolates positional embeddings to support inputs up to 980×980 and adds NaViT-style native-resolution handling. Native aspect ratios matter in manipulation because cameras often produce wide desktop views, and objects near the gripper may occupy long thin regions. Forcing every image into a square can distort relative geometry before the language model has even started reasoning.

The generation path uses the frozen FLUX VAE. It downsamples by a factor of eight and produces 16 latent channels. The VAE latents are reshaped so that 2×2 spatial patches become vectors, then projected into the language model's hidden dimension. A decoder reconstructs an image from the predicted latent. Keeping the VAE frozen avoids asking the training process to relearn image compression while the reasoning experts are still forming.

Actions use a much simpler interface. A linear layer maps continuous action dimensions into the model hidden space, and a symmetric linear decoder maps hidden states back to action coordinates. This is appropriate because the action vector is low-dimensional and already carries physical structure. Adding a heavy action tokenizer would risk losing precision and introducing unnecessary quantization into a system whose main challenge is semantic and visual reasoning.

Generating supervision for a multimodal chain of thought

EM-CoT training cannot be built from raw trajectories alone. A trajectory contains observations and low-level actions, but it does not naturally contain a task plan, a faithful subtask boundary, first-person reasoning, or a labeled visual goal. HALO therefore uses a three-stage automatic pipeline. First, rule-based matching converts continuous proprioception into motion primitives. The rules inspect displacement speed, gripper changes, idle intervals, and dominant movement directions. Second, Qwen3-VL annotates the trajectory. Third, the terminal frame of each subtask becomes the visual subgoal.

The rule-based stage is valuable because it uses physically measured action structure rather than asking a language model to infer timing from pixels. It can identify when a gripper remains idle long enough to split two motion segments, or when displacement is concentrated along one axis. The resulting boundaries are not perfect, but they anchor the learned narrative to actual robot behavior.

The Qwen3-VL stage is split into two passes. The first pass receives frames, per-arm motion descriptions, and optionally a global trajectory image, then produces a high-level ordered subtask sequence. The prompt explicitly discourages low-level phrases such as “reach” or “grasp” when they are merely motions, and instead asks for goal-oriented descriptions such as “pick up the red cup.” The second pass aligns each frame range to one subtask and writes first-person reasoning that connects visual observations, the overall goal, and the movement being executed.

The separation is sensible because task planning and frame alignment are different inference problems. A single prompt that asks a model to discover the plan and assign every frame simultaneously can produce inconsistent boundaries. Planning first gives the second pass a finite list of candidate stages. The second pass can then use the rule-derived action descriptions and the visual sequence to align frames and generate local reasoning.

thought = (
    f"<|subtask|>\n{subtask}\n<|\\ subtask|>\n\n"
    f"<|movement|>\n{movement}\n<|\\ movement|>\n\n"
)
goal_image = observation[key][goal_frame_idx]
actions = target_qpos[step_id:step_id + CHUNK_SIZE]

Source correspondence: data/robotwin_dataset.py builds a sample from subtask text, arm movements, a goal frame, and a future action chunk.

EM-CoT synthesis pipeline
Figure 2: Motion primitives provide temporal anchors, the vision-language annotator writes plans and reasoning, and subtask terminal frames provide visual goals.

The terminal-frame target is sparse by design. Dense future prediction would force the model to model every small appearance change, including frames where the robot is between meaningful states. Sparse stage targets reduce this redundancy and make the visual objective align with a decision boundary. The cost is that the model may learn only a few strategic snapshots rather than a continuous world model. Later limitations discussion returns to this tradeoff.

The two-stage training recipe

Stage one: build a broad multimodal and manipulation base

The first training stage combines three data families. LLaVA-NeXT-779k supplies general visual question answering and is optimized with cross-entropy loss. OXE robot trajectories and SSv2 egocentric videos supply visual future prediction, trained with a flow-matching mean-squared objective. OXE is also reused for direct action prediction, supervised by an L1 flow-matching loss. The visual branch uses both the semantic ViT representation and the VAE latent representation so that the model can reason about object identity while retaining spatially precise generation features.

The total objective places more weight on the embodied signals:

$$\mathcal{L}_{\text{pt}}=0.25\mathcal{L}_{\text{CE}}+0.5\mathcal{L}_{\text{MSE}}+\mathcal{L}_{\text{L1}}$$

The cross-entropy term preserves general language and visual grounding. The MSE term asks the model to predict scene evolution under an instruction, which is more than static recognition. The L1 term learns executable action chunks. The weights are not presented as a universal law; they are the recipe used in this work to stop the much larger general VQA corpus from overwhelming the manipulation data.

The paper trains this stage for 90,000 steps on 32 H100 GPUs. Sequence lengths and attention implementation are adjusted for the large multimodal batches, including Flex Attention in the training setup. The scale is substantial, but the conceptual point is that HALO does not begin reasoning from an action-only checkpoint. It first learns a model in which language understanding, visual dynamics, and control all have dedicated training pressure.

Stage two: teach the model to use its own intermediate outputs

The fine-tuning stage uses EM-CoT trajectories. Each sample contains a language instruction, history frames, a textual task plan, a visual subgoal, and the action chunk that moves the robot toward that goal. The joint objective is:

$$\mathcal{L}_{\text{ft}}=\mathcal{L}_{\mathbf{r}}+\mathcal{L}_{\hat{\mathbf{o}}}+\mathcal{L}_{\mathbf{a}}$$

Here $\mathcal{L}_{\mathbf{r}}$ supervises textual reasoning, $\mathcal{L}_{\hat{\mathbf{o}}}$ supervises the visual target, and $\mathcal{L}_{\mathbf{a}}$ supervises the action. VQA data is mixed into fine-tuning to reduce catastrophic forgetting. The model is not merely trained to produce an explanation after the action; all three losses are optimized in the same structured sequence, so the action expert learns to consume the intermediate context it will see at inference time.

Simulation fine-tuning uses 2,500 expert demonstrations from RoboTwin 2.0, covering 50 tasks with 50 demonstrations each, for 110,000 steps. Real-world fine-tuning uses 320 demonstrations from four tasks and runs for 80,000 steps. Inference observes three history frames and emits an action chunk of 16 for simulation or 50 for the real robot. The longer real-world chunk reduces decision frequency but increases the consequence of a bad chunk.

HALO training recipe
Figure 4: The first stage trains general visual generation, understanding, and action prediction. The second adds textual plans, movement descriptions, goal images, and action prediction in one EM-CoT format.
flowchart LR
  A[Instruction history] --> B[Understand scene]
  B --> C[Write subtask plan]
  C --> D[Predict goal latent]
  D --> E[Decode goal image]
  C --> F[Predict action chunk]
  E --> F
  B -. shared context .- D
  B -. shared context .- F
  C -. shared context .- F
    

The diagram is specific to HALO's dependency structure: the visual target is conditioned on the text plan, and the action expert reads both. The shared-attention links are shown as dotted connections because they operate on the context representation, not by collapsing the three output modalities into one head.

What RoboTwin 2.0 shows

RoboTwin 2.0 is a 50-task manipulation benchmark. Each task is evaluated 100 times in an Easy clean setting and 100 times in a Hard setting with domain randomization; the reported number is the average task success rate. HALO reaches 80.5% Easy and 26.4% Hard. π0 reaches 46.4% and 16.3%, so the absolute gains are 34.1 and 10.1 percentage points. The paper also reports relative gains of 73.5% and 62.0%, but the absolute differences are easier to interpret because they are not sensitive to the smaller baseline.

Table 1: Average success rate on RoboTwin 2.0, percent
MethodEasyHard
Diffusion Policy28.00.6
RDT-1B34.513.7
π046.416.3
HALO w/o EM-CoT75.321.2
HALO80.526.4

The table separates two sources of improvement. HALO without EM-CoT already reaches 75.3% Easy and 21.2% Hard, which is well above the strongest listed baseline. That result indicates that the mixture-of-transformers architecture and versatile pre-training provide a strong controller even when explicit reasoning is removed. Re-enabling EM-CoT adds 5.2 points Easy and 5.2 points Hard. The reasoning chain is therefore not responsible for the entire lead, but it is associated with a consistent additional gain in both environments.

Hard-setting performance deserves more attention than Easy performance. Diffusion Policy collapses to 0.6%, showing that a reactive imitation policy can be extremely brittle when visual randomization changes the input distribution. RDT-1B retains 13.7%, but the jump to 26.4% suggests that HALO's combination of general multimodal training, future visual prediction, and structured conditions is doing more than matching object appearance. The qualitative section argues that the model continues to identify the right target and plan when the background or object appearance changes.

The paper highlights gains on fine-grained tasks such as “Blocks Ranking Size” and “Stamp Seal,” where contact and ordering matter. Those examples are useful because they require more than moving toward the largest colored region in the image. They require representing a relationship between objects and checking whether the current state has completed the intended relation. A visual subgoal can make such a relation concrete, while textual reasoning can identify which relation is next.

Why each training source matters

The training-recipe ablation is unusually informative because the degradation is large. Removing visual-generation data changes Easy from 75.3% to 58.2% and Hard from 21.2% to 10.5%. The model still has a language and action base, but it has lost the explicit pressure to predict future visual states. Removing visual generation and textual VQA reduces performance to 42.9% Easy and 3.9% Hard. Removing all three leaves 32.4% Easy and 0.0% Hard.

Table 2: Training-recipe and reasoning ablations, percent
Training recipeEasyHardReasoning variantEasyHard
Full75.321.2Full EM-CoT80.526.4
w/o V58.210.5w/o textual reasoning77.818.3
w/o V+T42.93.9w/o visual subgoal76.122.5
w/o V+T+A32.40.0w/o both75.321.2

The EM-CoT ablation shows a more nuanced pattern. Removing textual reasoning reduces Hard from 26.4% to 18.3%, a larger loss than removing the visual subgoal, which gives 22.5%. Removing both returns to 21.2%. The exact ranking should not be overinterpreted without per-task variance, but it indicates that the text branch contributes task ordering and semantic disambiguation, while the visual branch contributes local spatial grounding. Neither is sufficient on its own.

The pre-training ablation also challenges a common assumption that a strong action corpus makes visual-language data optional. In this setup, the absence of all pre-training data leaves zero Hard success. The result supports the authors' claim that broad pre-training establishes competencies that downstream fine-tuning cannot synthesize from a small task-specific corpus. It does not prove that every component is necessary for every robot or task, but it does show a strong dependence in this benchmark.

Qualitative reasoning and real-world evidence

Figure 5 compares generated plans and goal images with ground-truth future states in clean and randomized RoboTwin scenes. The successful examples show the textual plan tracking the semantic stage while the visual target places the gripper and object in a spatially meaningful relation. Under randomization, the model sometimes preserves the correct object role even when colors, textures, and background differ from the clean demonstration. The paper uses this to argue that the control signal is not only matching a fixed pixel template.

EM-CoT qualitative results
Figure 5: Qualitative comparison of textual reasoning and predicted subgoal images in clean and unseen randomized environments.

The real-world setup uses a Cobot Magic Mobile ALOHA platform and only the head camera. Four tasks stress different capabilities: sweeping buttons with a tool, nesting cups with two arms, handing over a screwdriver between arms, and putting a lemon into a drawer. Each task contributes 80 trajectories, and all trajectories are processed into EM-CoT training samples. The evaluation adds visual distractors, light variation, background changes, and a novel broom replacement.

Real robot tasks and settings
Figure 6: Four real-world long-horizon tasks and the basic and generalization settings used for evaluation.

The paper reports that HALO outperforms π0 and π0.5 across the basic and generalization settings, with the baselines degrading more under visual shift. It does not provide a compact textual table of per-task success rates in the main paper, so a careful reader should not invent exact numbers from the plotted bars. The defensible claim is the reported ordering and trend: HALO has the highest success rate across the four tasks in the authors' evaluation, and the advantage is presented as evidence that EM-CoT supports real-world long-horizon execution.

Reading the released code

The repository exposes more than a conceptual demo. inference/model.py contains the Qwen2.5 backbone, SigLIP2 visual encoder, FLUX VAE, action expert configuration, and inference settings. It uses an approximately 20% probability of enabling textual subtasks and goal images, otherwise the action path runs without the full reasoning chain. inference/halo_inferencer.py implements the text-to-image-to-action sequence and supports returning either a decoded visual goal or its latent. data/robotwin_dataset.py aligns the three-frame history, subtask text, goal frame, and action chunk. train/pretrain_unified_navit.py combines the language, visual, and action losses and configures the three-expert model.

This source structure supports several claims in the paper. The action output is genuinely continuous and decoder-based rather than a simple action token class. The visual branch has a separate latent path and VAE. The fine-tuning sample contains both intermediate supervision and the corresponding action block. The code also exposes the practical inference shortcut: the full chain is sampled stochastically, so deployment does not necessarily pay for textual and visual generation on every control step.

The shortcut should be treated as a deployment fact, not a contradiction. Training teaches the model to model the full conditional sequence; inference can choose a smaller computation path. If the task is familiar and the action head is already confident, skipping the explicit goal may preserve performance while reducing latency. If the scene is novel, enabling the full chain may provide the reasoning benefit observed in the benchmarks. The paper does not systematically report the success-latency curve for this choice.

Limit 1: automatic annotation can create confident but wrong reasoning

The data pipeline relies on rule-based segmentation and a Qwen3-VL annotator. Rule-based primitives are transparent, but they are not a complete physical event detector. If an object slips without a clear idle interval, if a gripper closes without moving an object, or if contact is hidden by the wrist, the extracted primitive boundary may be wrong. The language model then receives a misaligned frame list and is asked to produce a coherent subtask sequence. Fluency is not the same as correctness: the annotator can generate a logically consistent narrative around a boundary that does not match the actual interaction.

Once an erroneous narrative becomes training data, the textual expert learns both the correct and incorrect reasoning patterns. The visual expert is supervised by the terminal frame of the erroneous stage, so it may learn to predict a scene that belongs to a different subtask. The action expert then conditions on mutually inconsistent text and image targets. The resulting policy may still fit some trajectories because the action loss penalizes bad behavior, but the intermediate reasoning objective can reward explanations that are semantically plausible yet causally false. The paper does not report an independent human audit of annotation accuracy or a correction model for uncertain segments.

This risk is especially relevant in multi-arm tasks. A subtask can involve coordinated motion from both arms, while the rule-based system and prompt describe each arm separately. If the temporal alignment chooses a frame where one arm has reached its target but the other is still moving, the visual goal may depict a partial relation. The first-person reasoning may then say the subtask is complete because the visible status of the dominant arm looks correct. The model needs a notion of joint completion, not just an individually plausible arm trajectory.

Limit 2: real-world coverage is evidence, not proof of broad generalisation

The real-world evaluation uses one mobile bimanual platform, four tasks, 80 demonstrations per task, and a head camera as the only visual input. That is a meaningful step beyond simulation, but it is still a narrow slice of embodied manipulation. The experiments do not cover different kinematic structures, tactile feedback, wrist cameras, transparent or deformable objects, human-robot collaboration, or long sessions with compounding drift. A policy that succeeds on a fixed tabletop and a known set of objects may still fail when camera calibration, gripper geometry, or contact mechanics change.

The single head-camera choice also changes the meaning of visual reasoning. A global view can provide task context, but it may not resolve the final millimeters of an insertion, a screwdriver handover, or a grasp hidden by the gripper. HALO's textual plan and visual subgoal can help disambiguate the scene, yet the action expert has no independent local sensing stream. The paper's strongest claim should therefore be scoped to the evaluated platform and its visual setting. The results support a promising direction, not a universal guarantee of cross-embodiment transfer.

The evaluation protocol also reports success rates without a detailed analysis of failure recovery. A deliberative controller should ideally detect when its predicted subgoal has become stale and replan. HALO regenerates the chain at the next policy step or chunk boundary, but the paper does not separate failures caused by a bad plan, a bad visual goal, an unreachable goal, and a good plan followed by poor low-level control. Without that decomposition, it is difficult to know which part of EM-CoT deserves further scaling.

Limit 3: the full reasoning chain has an unreported latency cost

Generating a visual subgoal with a diffusion or flow process and then running action denoising is not free. The paper reports strong success rates but does not provide a systematic latency table for text generation, visual generation, action generation, and end-to-end deployment. It also does not report how success changes when the full chain is used on every control step. The released inference code defaults to enabling the full textual and visual path with probability around 0.2, which suggests the authors themselves are aware of a runtime tradeoff.

For a real robot, latency interacts with safety and stability. A delayed action chunk may be executed against a scene that has already changed, and a delayed visual goal may condition the controller on an obsolete target. The paper's sparse subtask targets make the representation useful, but they do not solve the systems problem of scheduling multimodal generation under a control deadline. A production system would need confidence estimates, asynchronous planning, or a fallback action policy when the reasoning stage misses its budget.

The absence of a latency breakdown also weakens comparisons to reactive VLA baselines. If HALO uses substantially more compute per decision under the full chain, its success advantage should be evaluated alongside throughput and hardware occupancy. The 20% sampling strategy is an important clue, but the paper does not show which task states trigger the full chain in deployment or whether the trigger is learned from uncertainty. This is a promising engineering question rather than a flaw in the central representation idea.

Limit 4: a sparse visual target is not yet a complete world model

The visual goal predicts a terminal frame for the current subtask. It offers spatial guidance, but it does not explicitly model contact forces, object mass, friction, or multi-step physical feasibility. A predicted image can look plausible while representing a state the robot cannot reach with the available kinematics. The action expert must learn the gap between visual feasibility and physical feasibility from action supervision. In a novel layout, that gap may be exactly where the policy fails.

Similarly, the text reasoning can become post hoc language that correlates with the correct action without being the reason the action is chosen. The ablations show that removing the text branch hurts performance, which is evidence of usefulness, but they do not identify which words or concepts change the action. A stronger evaluation would intervene on the generated plan, counterfactually change the goal image, and measure whether the action follows the selected intermediate state. The present experiments establish an association between multimodal reasoning and control gains, not a fully mechanistic explanation.

Assessment: what is genuinely new

The strongest contribution is the architectural treatment of heterogeneous intermediate outputs. HALO does not merely add a language explanation beside an existing VLA, nor does it generate goal images as a separate auxiliary task. It defines a single dependency sequence in which the image is generated from the text, and the action is generated from both. The mixture of experts provides a plausible mechanism for keeping those computations distinct, while the shared attention and control tokens prevent them from becoming isolated models. The training recipe then supplies aligned supervision for every edge in that graph.

The empirical results are also stronger than a simple “reasoning helps” anecdote. The paper removes textual reasoning and visual subgoals separately, showing that each branch changes performance. It removes visual, language, and action pre-training sources progressively, showing that Hard performance degrades sharply as the foundation narrows. It evaluates a real bimanual robot under several distribution shifts. Even with the limited task set, this combination gives a more useful picture than one qualitative rollout.

The paper's most important open question is how to make the intermediate chain trustworthy under deployment. Automatic annotation introduces semantic and temporal uncertainty. The visual target can be plausible but physically unreachable. The full chain can be too slow for a control loop and is sampled only occasionally in the released inference code. These issues do not erase the reported gains, but they define the next engineering frontier: uncertainty-aware annotation, physical feasibility checks, and an explicit latency policy for when deliberation is worth its cost.

Conclusion

HALO is a well-motivated attempt to merge the semantic strength of textual reasoning with the spatial specificity of visual foresight. Its three experts preserve modality-specific generation, while shared self-attention and structured masks make the experts cooperate in one policy. The automatic pipeline converts raw robot trajectories into aligned plans, reasoning, visual goals, and action chunks, and the two-stage training recipe gives the model both a broad multimodal foundation and task-specific deliberation.

On RoboTwin 2.0, the full model reaches 80.5% Easy and 26.4% Hard success, improving over π0 by 34.1 and 10.1 percentage points. The ablations show that both textual reasoning and visual subgoals contribute, and that the visual-generation and language data used in pre-training are not expendable in this benchmark. Real-robot experiments on four long-horizon tasks provide supporting evidence on a bimanual mobile manipulator.

The next version of this line of work should treat the reasoning chain as something to verify, not only something to generate. A useful system will audit its annotations, test whether predicted goals are physically reachable, replan when the visual state becomes stale, and schedule expensive deliberation only when uncertainty justifies it. HALO supplies a credible architecture for that direction: the intermediate states are already present, shared across experts, and exposed to control. The remaining challenge is making them reliable enough to supervise in the wild.

Key sentence: HALO's real proposal is not that a robot should produce more text before acting, but that task meaning, imagined future state, and continuous control should be coupled through one inspectable generative path.

Related Papers

mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs

mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs

mimic-video couples an Internet-scale pretrained video model with a flow-matching action decoder. Latent visual plans carry physical dynamics, while the decoder acts as an inverse dynamics model that outputs robot actions. Evaluations on SIMPLER-Bridge, LIBERO, and real-world bimanual manipulation show stronger performance with far less action data and faster convergence.

Jonas Pai, Liam Achenbach, Victoriano Montesinos · 111 views · Dec 17, 2025
Video-Action ModelRobot Manipulation
Dec 17, 2025
OmniBrain-0: A Unified Agentic Model for Embodied Intelligence

OmniBrain-0: A Unified Agentic Model for Embodied Intelligence

General-purpose embodied intelligence requires robots to coordinate diverse skills over long horizons, track task progress, and recover from failures. Specialist policies alone do not ensure task-level reliability, while powerful multimodal agents rely on costly reasoning and external experience that compact models struggle to reproduce. We introduce OmniBrain-0, a lightweight unified embodied agent that coordinates manipulation and navigation through a shared high-level decision model, integrating task understanding, planning, progress tracking, and feedback-driven recovery. Our progressive training framework combines embodied pretraining, domain-specialized agent distillation, and multi-teacher on-policy distillation (MOPD). It distills powerful teachers' interaction experience into manipulation and navigation experts, then consolidates their decision capabilities into OmniBrain-0. Unification therefore occurs within the learned decision model rather than only at the tool interface. We further introduce Rubric-Guided Recursive Self-Improvement (RSI), a data flywheel in which a Meta-Agent turns execution evidence into refined rubrics, step-level reminders, and verified trajectories for domain-expert updates before their reconsolidation into OmniBrain-0 through MOPD. We evaluate OmniBrain-0 on LIBERO-Pro, LIBERO-Plus, RoboCasa365, RoboTwin 2.0, and R2R-CE Val-Unseen, as well as multiple real-world manipulation tasks. OmniBrain-0 combines strong cross-domain task performance with substantially lower inference and execution overhead than its closed-source teacher, demonstrating efficient transfer and consolidation of embodied capabilities into a compact unified agent.

Xiaomi Embodied Intelligence Team, Yuechen Luo, Chong He · 434 views · Oct 26, 2026
Embodied AIVLA
Oct 26, 2026
GRAFT: Grounded and Efficient Online Reinforcement Adaptation for Fine-Grained Robot Manipulation

GRAFT: Grounded and Efficient Online Reinforcement Adaptation for Fine-Grained Robot Manipulation

Pretrained vision-language-action (VLA) policies provide strong priors for robot manipulation, yet adapting them online to fine-grained biomedical tasks remains challenging. Task success often hinges on subtle, view-dependent visual cues, while task-level rewards provide little guidance about which regions matter, making it difficult to learn task-relevant visual grounding from limited real-robot interaction. Online adaptation is further constrained by the computational cost of VLA inference and replay-based updates. We introduce GRAFT (Grounded Reinforcement Adaptation for Fast Task Learning), a framework for efficient online VLA adaptation through grounded perception. GRAFT uses region-level supervision to learn view-specific visual anchors that focus perception on task-relevant local cues without requiring region proposals at deployment. It further combines single-step action generation with cached visual-language prefix reuse to accelerate online learning. Across four biomedical manipulation tasks, GRAFT improves success rates by 32.5 percentage points under matched adaptation budgets, while reducing the computational overhead of online policy updates.

Yibo Qiu, Haoliang Ye, Shu'ang Sun · 330 views · Aug 27, 2026
VLARobot Manipulation
Aug 27, 2026
SERF: Spatiotemporal Environment and Robot Feature Map for Long-Horizon Mobile Manipulation

SERF: Spatiotemporal Environment and Robot Feature Map for Long-Horizon Mobile Manipulation

SERF represents the environment and the articulated robot body as neural points in one shared latent space and updates them online. Environment points follow object-level rigid registration, while robot surface points are posed by forward kinematics. A shared residual decoder maps local point features to DINOv3 patch embeddings, and a two-stage Point Transformer produces eight map tokens from three robot-centered radii, both end effectors, robot-only, environment-only, and global branches. These tokens condition a VLA policy together with images, proprioception, and language. Across three long-horizon BEHAVIOR-1K tasks, SERF raises average progress from 44.0% for fine-tuned PI0.5 to 58.7%, and the paper also reports gains when labels and base poses are estimated onboard, when object-drop recovery is required, and in unvisited regions.

Sunghwan Kim, Byeonghyun Pak, Kehan Long · 304 views · Jun 11, 2026
VLAMobile Manipulation
Jun 11, 2026

As an Amazon Associate, we earn from qualifying purchases.