PAPER DEEP DIVE
XBot-Omni: An Omni-Modal Model for Embodied Understanding and Interaction
Embodied interaction requires an agent to integrate multimodal perception with verbal and nonverbal behaviors to effectively communicate its intentions, including speech, body motion, and facial expressions. We present XBot-Omni, a Thinker–Talker–Actor architecture that extends the Omni paradigm to robotic modalities, including body motion and facial expressions, with ``X'' representing the extensibility to robotic-specific modalities. The Thinker processes text, images, audio, and video to formulate contextually appropriate responses, which are then realized by the Talker and Actor through speech and coordinated bodily expressions, respectively. In this report, we innovatively introduce the concept of ``Actor'' as a dedicated component for embodied agents: a coordinated, streaming module that leverages the native representations of the Thinker and Talker within an end-to-end pipeline, introducing only modest latency overhead. This unified architecture enables multimodal generation conditioned on omni-modal inputs directly to prevent information loss in staged designs. We comprehensively evaluate XBot-Omni across seven benchmarks, demonstrating strong multimodal understanding capabilities on both general-domain and robot-centric videos, as well as high-quality, natural, and well-coordinated motion generation. Furthermore, our results validate its capability to generate embodied multimodal responses directly from omni inputs, meeting the diverse requirements of embodied interaction. XBot-Omni enables robots not only to perceive the omni world but also to express themselves through speech and bodily behaviors, extending Omni models from conversational interaction to embodied communication in physical environments.
Central claim
XBot-Omni joins multimodal reasoning, streaming speech, body motion, and facial expression in one Thinker-Talker-Actor pipeline. The architectural bet is that embodied communication is better generated from shared native representations than from a sequence of text, waveform, and motion APIs.
Code status: the project page currently contains only a coming-soon notice. The linked GitHub site repository is a placeholder and provides no model code, checkpoints, training configuration, or inference API. This analysis therefore makes no claim of source-level verification. The reported results cannot be independently reproduced from the public release.
Why an embodied omni model needs more than speech
An embodied response is not merely an answer with a voice attached. If a robot says, "the red cup is on your left," its gesture must refer to the correct object, begin near the relevant phrase, and remain physically plausible from the robot's current posture. Dialogue context, scene structure, spatial reference, and action timing all contribute to the communicative act. XBot-Omni treats these factors as parts of one response-generation problem rather than independent downstream tasks.
General omni-modal models have made substantial progress in combining text, vision, and audio. They can describe a scene, answer a temporal question, and synthesize speech. Co-speech gesture systems have likewise advanced motion realism and semantic control. Their assumptions remain different, however. General omni models usually stop at language or audio output, while gesture systems usually receive text, transcript, or synthesized speech after the response has already been formed. The missing interface is a representation that still contains multimodal intent and scene reference when motion generation begins.
A conventional cascade exposes a clean API: generate a response, synthesize speech, then drive motion from the audio. It also discards information at each boundary. Acoustic features need not preserve the visual referent or the internal interpretation that caused a phrase to be selected. The motion module may wait for waveform release and then recover speech properties through another encoder. This is manageable for offline animation but unattractive for a robot that should begin behaving while speech is still being generated.
The paper's alternative is to expose native states from the Thinker and Talker. Native here means hidden representations inside the generation process, before they are reduced to text or a waveform. Thinker states carry contextual semantics and visually grounded intent. Talker states carry evolving speech content and progress. The Actor consumes both through adapters and combines them with explicit gesture events and its own motion history.
This design also introduces a new coupling cost. A motion branch trained against one upstream representation may be sensitive to later model replacement, quantization, or distillation. The paper demonstrates a coordinated system, but it does not report transfer experiments after changing the Thinker or Talker backbone. The architectural benefit and the maintenance risk should therefore be assessed together.
System organization
Figure 1 presents X2 and Q1 interaction examples involving speech, facial behavior, body movement, and pointing. These images establish that the paper targets physical humanoid platforms, not only avatar animation. They do not expose enough detail to reproduce deployment, calibration, timing, or failure handling.
The Thinker maps text, images, audio, video, and dialogue history into a shared response context. The Talker converts that response into streaming speech tokens and exposes hidden states during autoregressive generation. The Actor uses a semantic controller, a speech-action aligner, an autoregressive body-motion generator, and a separate flow-matching facial branch. The body and face are different generative mechanisms placed on a common response timeline.
flowchart LR
OBS["Text image audio video
and dialogue context"] --> THINK["Thinker
multimodal reasoning"]
THINK --> CONTEXT["Response meaning
scene reference"]
THINK --> TALK["Talker
streaming speech model"]
TALK --> TOK["Speech tokens"]
TOK --> DEC["Speech decoder"]
TALK --> HID["Native speech states"]
CONTEXT --> CTL["Semantic Action Controller"]
CTL --> EVENT["Gesture event
category attributes timing"]
CONTEXT --> ALIGN["Speech Action Aligner"]
HID --> ALIGN
HID --> MOTION["Motion Generator"]
EVENT --> MOTION
ALIGN --> MOTION
MOTION --> BODY["Body motion tokens"]
BODY --> KIN["X2 or Q1 kinematics"]
CONTEXT --> FACE["Chunk Transformer"]
HID --> FACE
FACE --> FLOW["Conditional flow matching"]
FLOW --> FEX["Facial expression"]
DEC --> RESP["Embodied response"]
BODY --> RESP
FEX --> RESP
The important distinction from a generic diagram is the two-way use of semantics. The Thinker does not merely produce a text response that later gets converted into motion style. Its hidden context becomes an explicit set of gesture requirements. Similarly, the Talker is not only an audio renderer. Its internal states condition motion before the final waveform exists.
Thinker: temporal multimodal context
The Thinker receives modality-specific encodings and combines them in a shared backbone. Text is tokenized directly. Images and video frames pass through a vision encoder. Audio is resampled and converted to mel-spectrograms, processed by an AuT initialized from Qwen3-Omni, and projected into the backbone representation. The system then reasons over the joint sequence and dialogue history.
Video and audio-visual inputs receive formatted textual timestamps in seconds before each temporal patch. The timestamp is explicit content in the sequence, not only a positional index. This matters because positional magnitudes and spacing can change with sequence length and frame rate. A common textual clock gives acoustic and visual events a stable reference for questions about order, duration, and alignment.
The training procedure has three stages. Supervised fine-tuning establishes a shared base for perception, temporal reasoning, reference resolution, speaker and addressee identification, and proactive interaction decisions. The paper then trains multiple domain-specific experts with GRPO using rule-based rewards or an LLM judge. Finally, multi-teacher on-policy distillation consolidates these frozen experts into one student.
The distillation objective is defined on the student's own generated trajectory. For each task domain, the student samples an input and response, and the corresponding expert supplies token-level supervision at prefixes reached by the student:
$$ \mathcal{L}_{\mathrm{MOPD}}(\theta)= \sum_{k=1}^{K}\alpha_k \mathbb{E}_{x\sim\mathcal{D}_k} \mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)} \left[ \frac{1}{|y|}\sum_{t=1}^{|y|} D_{\mathrm{KL}} \left( \pi_\theta(\cdot\mid x,y_{<t}) \middle\| \pi_{\phi_k}(\cdot\mid x,y_{<t}) \right) \right]. $$
Here $\alpha_k$ is the probability of sampling task domain $k$, $\phi_k$ are fixed expert parameters, and $\theta$ is the trainable student. The on-policy construction is useful because supervision covers prefixes produced by the student rather than only the human reference trajectory. It does not prove that all expert capabilities survive consolidation, and no per-domain forgetting analysis is reported.
Talker: speech progress as a motion signal
The Talker follows an autoregressive streaming speech design. A hybrid mixture-of-experts temporal backbone predicts speech-token progression, while a depth predictor performs multi-token prediction over residual codebooks. A speech codec decodes the generated tokens incrementally. This allows audio to emerge without synthesizing the complete response in one pass.
XBot-Omni adds an interface from the Talker's evolving hidden states to the Actor. The speech branch used for motion can therefore consume representations before waveform decoding and without a second audio encoder. In principle, this avoids waiting for a stable audio file and keeps speech-generation state close to motion generation.
The latency claim needs precise scope. The paper reports a median first-motion latency of 527 ms in its demonstration. That is not a full latency distribution, and it does not establish synchronization under queuing, network delay, decoder variation, or repeated long responses. Native-state access removes one source of cascade delay; it does not remove the need to predict speech progress, schedule events, and generate physically continuous motion.
Body motion with semantic control
Let $H^{\mathrm{th}}=\{h_i^{\mathrm{th}}\}$ and $H^{\mathrm{sp}}=\{h_n^{\mathrm{sp}}\}$ denote the native Thinker and Talker state streams. A causal recurrent adapter maps speech states into the motion-conditioning space. At motion step $t$, the generator predicts a tuple of discrete motion codes from previous motion codes, adapted speech states, and active event conditions:
$$ p_\theta(Z\mid H^{\mathrm{sp}},U) =\prod_t p_\theta\left(z_t\mid z_{<t},e_{\le t},U_t\right). $$
$z_t$ contains the motion codes, $e_{\le t}$ is the available adapted speech history, and $U_t$ represents active semantic and temporal conditions. When the event is a point, $U_t$ also carries the target position in the Actor's body-centric frame. Generated codes are decoded into articulated motion and remain available as history, so a later requirement can begin while earlier movement is still unfolding.
The Semantic Action Controller converts response intent into explicit events. Each event contains a gesture category $y_e$, semantic parameters $a_e$, an availability mask $m_e$, and the associated response span. The event condition is formed as:
$$ c_{e,t} =\left[ \phi_y(y_e); \phi_a(a_e\odot m_e,m_e); \phi_\tau(b_{e,t},\delta_{e,t}) \right]. $$
$\phi_y$ encodes the category, $\phi_a$ encodes supported attributes such as hand, direction, or three-dimensional target, and $\phi_\tau$ encodes the start signal and elapsed time. The mask prevents unavailable attributes from being interpreted as meaningful zero values. This separation lets the generator learn how to execute motion while the controller specifies what the motion should communicate.
Multiple events can be active at once. The paper uses additive modulation around an empty condition $c_\emptyset$:
$$ (\gamma_t,\beta_t) =g(c_\emptyset) +\sum_{e\in A_t}\left[g(c_{e,t})-g(c_\emptyset)\right], \qquad \tilde v_t=(1+\gamma_t)\odot v_t+\beta_t. $$
This formulation represents event composition as offsets from the same baseline and avoids enumerating every subset of simultaneous gestures. It does not solve physical conflicts. If two events demand incompatible targets or hands, the additive condition has no explicit constraint solver; the paper does not report conflict-specific failure rates.
The Speech-Action Aligner estimates correspondence between speech positions and response-text positions. For position scores $\ell_{n,i}=\alpha q_n^{\top}k_i$, a forward recursion combines emission evidence with transition probabilities:
$$ \pi_n(i) \propto \exp(\ell_{n,i}) \sum_r A_n(r,i)\pi_{n-1}(r). $$
For an event $e$, the scheduler estimates when its reference phrase becomes active and backs off by the expected gesture duration:
$$ \widehat T_e =\min\{T_n:n\in O,\widehat p_n\ge i_e\}, \qquad T_e^{\mathrm{start}} =\max(T_e^{\mathrm{next}},\widehat T_e-d_{y_e}). $$
$\widehat p_n$ is the predicted speech position, $i_e$ is the event's text position, and $d_{y_e}$ is the expected duration. The start condition tries to prepare before the relevant word and avoid overlap with the previous event. Errors in speech progress, target position, or duration propagate into the motion schedule.
A separate facial pathway
Facial behavior uses a continuous generator rather than the discrete autoregressive body-token path. A Chunk Transformer forms a temporal condition $C_j$ from Thinker and Talker states. Given a target facial chunk $F_j^\star$ and Gaussian noise $\epsilon_j$, the model trains a conditional flow with:
$$ F_j(\tau)=(1-\tau)\epsilon_j+\tau F_j^\star, $$
$$ \mathcal{L}_{\mathrm{face}} =\mathbb{E}_{j,\tau,\epsilon_j} \left[ \left\| v_\eta(F_j(\tau),\tau,C_j) -(F_j^\star-\epsilon_j) \right\|_2^2 \right]. $$
The velocity field $v_\eta$ learns to move from noise toward a real facial chunk under response and speech context. Body and face remain different generative processes but receive shared semantic and speech information. The paper does not quantify body-face synchronization separately, so the coordination claim should be read as an architectural property rather than a measured multimodal timing result.
Data construction and staged training
The training corpus combines public embodied activity datasets with private human-human and human-robot recordings. Public sources contribute egocentric daily activity, manipulation, tool use, and procedural tasks. Private recordings add natural conversation, social exchange, collaboration, and robot-assisted tasks with synchronized speech, environmental audio, visual observations, gesture, and facial behavior.
Annotation is managed by an agent that coordinates speech recognition, speaker diarization, object detection and tracking, action recognition, and audio-visual event localization. The pipeline creates timestamped interaction episodes and generates questions and instructions over scene understanding, temporal relations, referents, and intent. Cross-modal consistency checks and evidence-based verification filter unreliable samples. This description is important but not externally auditable: the paper does not report annotation agreement, tool versions, rejection rates, or the effect of annotation errors on downstream failures.
Actor training begins by adapting fixed Talker states to action conditioning. Audio-derived features provide offline supervision. For adapter $A_\psi$, valid paired step set $\mathcal V$, target $e_n^\star$, and feature standard deviation $\sigma_e$:
$$ \mathcal{L}_{\mathrm{adapt}}(\psi) =\frac{1}{|\mathcal V|d_e} \sum_{n\in\mathcal V} \left\| \frac{A_\psi(h_{\le n}^{\mathrm{sp}})-e_n^\star}{\sigma_e} \right\|_2^2. $$
The Talker and motion codec remain fixed. This isolates the question of whether native speech states contain usable motion information from the question of whether the motion generator can exploit them. A separate alignment loss uses confidence-weighted speech-position labels:
$$ \mathcal{L}_{\mathrm{align}} =-\frac{1}{|\mathcal S|} \sum_{s\in\mathcal S}\frac{1}{T_s} \sum_{n=1}^{T_s}w_{s,n} \log P_{s,n}(i_{s,n}^\star). $$
Semantic-event training then combines natural co-speech motion with controlled gesture segments. The motion objective is a group-balanced cross-entropy over reference tokens:
$$ \mathcal{L}_{\mathrm{motion}} =-\sum_{g\in\mathcal G}\frac{\alpha_g}{N_g} \sum_{(t,k)\in g}m_{t,k} \log p_\theta \left(z_{t,k}^\star\mid\mathcal H_{t,k},e_{\le t},U_t\right). $$
A balanced completion loss predicts when the expressive core is complete and when motion can release into a transition. The body objective is $\mathcal L_{\mathrm{Actor}}=\mathcal L_{\mathrm{motion}}+\lambda_{\mathrm{comp}}\mathcal L_{\mathrm{comp}}$. The third stage replaces reference prefixes with model-generated history so that later states contain the system's own errors:
$$ \mathcal{H}_{t,k}^{\mathrm{gen}} =\left(\widehat z_{t,1:K},z_{t,k}^\star\right). $$
Earlier codebooks in the current step remain teacher-forced while preceding steps come from the model. Completion labels are masked when a generated trajectory makes the reference completion time unjustified. This is a sensible robustness measure, although the paper does not quantify how much exposure to generated history changes stability or naturalness.
From visual grounding to robot pointing
SGPT extends a semantic event with a scene-derived spatial target. A referring expression identifies an object; grounding, depth estimation, and camera intrinsics recover a point; the camera-to-body transform places it in the Actor frame:
$$ \widehat{\mathbf p}^{\,\mathrm{body}} =R_{\mathrm{cam}}^{\mathrm{body}} \mathcal P(\widehat g,\widehat D,K) +\mathbf t_{\mathrm{cam}}^{\mathrm{body}}. $$
$\widehat g$ is the grounding result, $\widehat D$ is estimated depth, $K$ contains intrinsics, and the rotation and translation define camera-to-body geometry. The target becomes part of $a_e$, while the referring phrase anchors the event in speech. The generator produces reference motion, and a platform-specific adapter converts it into X2 or Q1 joint execution using coordinate transforms, calibrated pointing axes, kinematics, and continuity constraints.
Task-conditioned training mixes pointing demonstrations with natural co-speech examples:
$$ \mathcal{L}_{\mathrm{PA}}(\theta_A) =\rho\mathbb E_{d\sim\mathcal D_P} \left[\mathcal L_{\mathrm{Actor}}(\theta_A;d,U^\star(d))\right] +(1-\rho)\mathbb E_{d\sim\mathcal D_N} \left[\mathcal L_{\mathrm{Actor}}(\theta_A;d,U^\star(d))\right]. $$
The Thinker and Talker remain frozen while Actor-side parameters adapt. The separation keeps upstream understanding and speech generation fixed, but it also means the system relies on the existing representations to expose the target correctly. Grounding, depth, camera calibration, and inverse-kinematics failures are all potential bottlenecks, and none is isolated by an end-to-end task-success study.
Evaluation: understanding and multi-turn reference
Daily-Omni measures six dimensions of audiovisual understanding over 30-second and 60-second videos. XBot-Omni reaches 86.30% overall accuracy against 84.74% for Qwen3.5-Omni-Plus. It leads every capability and both duration groups. The largest gains are in audiovisual alignment and event sequencing, which is consistent with the use of explicit audio-visual timestamps and temporally structured embodied data.
| Daily-Omni metric | XBot-Omni | Qwen3.5-Omni-Plus | Difference |
|---|---|---|---|
| Overall | 86.30 | 84.74 | +1.56 |
| Audio-visual alignment | 87.39 | 84.42 | +2.97 |
| Event sequence | 86.60 | 83.95 | +2.65 |
| 30-second group | 84.39 | 83.41 | +0.98 |
| 60-second group | 88.55 | 86.36 | +2.19 |
ReTurn asks a model to retain and revise references across five conversational turns. XBot-Omni scores 84.00% overall against 70.04% for Doubao-Seed-2.1-Lite. The advantage is especially large on Reground and Rebind, where an earlier interpretation must change. The model is weaker on Reconfirm and Retrieve, so the overall gain is not uniform accuracy across every interaction behavior.
| ReTurn task | XBot-Omni | Doubao-Seed-2.1-Lite | Interpretation |
|---|---|---|---|
| Overall | 84.00 | 70.04 | +13.96 points |
| Reconfirm | 87.20 | 98.24 | -11.04 points |
| Reground | 67.20 | 29.60 | +37.60 points |
| Retrieve | 93.12 | 97.28 | -4.16 points |
| Rebind | 88.48 | 55.04 | +33.44 points |
EgoProx reports intention, exploration, and exploitation scores of 65.54, 62.57, and 69.43. The exploitation relative score is 57.72, which is 0.14 below the strongest baseline. ERIBench adds another mixed picture: XBot-Omni scores 85.51 on intent inference, below Qwen's 91.45, but records 68.76 on safety boundaries, 59.17 on spatial relations, and 69.81 on grounding. Spatial relations improve by 20.99 points, while intent remains a weaker dimension.
Evaluation: motion, semantic quality, and timing
Motion quality is measured on 499 valid BEAT2 speech samples from 500 transcript excerpts. The semantic evaluation contains 119 valid short responses across ten targets: four directions, two extents, and four interaction gestures. A visual model observes anonymized, silent, front or oblique clips and assigns a score from zero to three. Overall semantic quality is:
$$ \mathrm{SQ} =\frac{1}{|\mathcal C|} \sum_{c\in\mathcal C} \left( \frac{1}{|\mathcal I_c|} \sum_{i\in\mathcal I_c}q_i \right). $$
Conditional timing uses only responses with a clear compatible expression and valid alignment:
$$ \mathrm{CT} =\left( \frac{1}{|\mathcal E|} \sum_{i\in\mathcal E}h_i \right)\times100\%. $$
XBot-Omni achieves an FGD of 1.351, lower than EMAGE at 1.389, and an overall SQ of 1.383 against 0.608 for Semantic Gesticulator. Direction, extent, and interaction scores are 1.750, 1.000, and 1.208. Fifty-four of 119 responses enter the clear compatible set, and 46 of those are timely, yielding 85.19% CT.
| Method | FGD lower is better | Overall SQ / 3 | CT |
|---|---|---|---|
| EMAGE | 1.389 | 0.467 | 70.59% (12/17) |
| Semantic Gesticulator | 9.817 | 0.608 | 56.52% (13/23) |
| XBot-Omni | 1.351 | 1.383 | 85.19% (46/54) |
The CT denominator is the main interpretive trap. XBot-Omni has 54 responses that satisfy the timing evaluation's eligibility conditions, while compared methods have only 5 to 23. The paper explicitly states that CT is neither an overall interaction success rate nor a paired comparison on an identical subset. The defensible conclusion is that XBot-Omni produced more clearly evaluable semantic gestures in this controlled cohort and aligned a high fraction of them, not that 85.19% of all embodied interactions succeed.
Evaluation: spatial grounding and pointing
TraceSpatial-Bench evaluates 2D and 3D start and end localization with a 20 cm 3D tolerance. XBot-Omni reaches 67.00% and 68.00% for 3D start and end, improving over the strongest corresponding baselines by 24 and 29 points. Its 2D scores are 47.00% and 53.00%, below the strongest 2D baseline. On SUN RGB-D, its five-category mAP at IoU 0.25 is 35.04% against the best baseline's 34.15%. The advantage is therefore concentrated in 3D target localization, not generic 2D perception.
For pointing, 100 public development clips with high human-reference scores are used. Score b measures spatial accuracy, Score a measures temporal alignment, and STG combines both. SGPT raises Score b from 0.357 to 0.990, Score a from 0.654 to 0.851, and STG from 0.241 to 0.843. Spatial accuracy nearly reaches the human reference of 0.999, while temporal alignment remains below 0.985.
| Pointing setting | Score b spatial | Score a temporal | STG joint | FGD | Beat alignment |
|---|---|---|---|---|---|
| Without SGPT | 0.357 | 0.654 | 0.241 | 0.485 | 0.432 |
| XBot-Omni with SGPT | 0.990 | 0.851 | 0.843 | 0.498 | 0.423 |
| Human reference | 0.999 | 0.985 | 0.984 | 0.389 | 0.393 |
SGPT improves target direction and speech timing, but it is not free. Pointing-arm FGD increases from 0.485 to 0.498, and beat alignment decreases from 0.432 to 0.423. Jerk improves from 0.558 to 0.530, and velocity diversity moves from 0.696 to 0.650, closer to the human value of 0.618. The result is a meaningful control tradeoff: stronger spatial semantics can reduce some distributional and rhythmic motion scores.
Limitations and deployment questions
The authors disclose that semantic CT uses method-specific eligible subsets and machine-assisted evaluation on controlled short responses. No human preference study is reported. This limits claims about naturalness, social appropriateness, and overall interaction success. A method can produce more clearly evaluable gestures and still fail in a long, noisy, multiparty interaction.
There is no dedicated limitations section and no public implementation. Model code, weights, training data statistics, expert count, optimization details, and robot calibration are unavailable. Reproduction is therefore impossible from the current release. This is especially consequential for an architecture whose central claim depends on native-state interfaces, motion tokenization, and platform-specific inverse kinematics.
The evaluation is also component-wise rather than end-to-end. Understanding, motion quality, spatial localization, and pointing are measured separately. The paper does not report a unified closed-loop task in which errors in grounding, depth, scheduling, motion generation, and physical execution interact. It cannot yet answer whether a grounding error is visible in the final gesture, whether speech jitter breaks timing, or whether pointing leads to successful object interaction.
Finally, robustness is undercharacterized. The 527 ms first-motion latency is a median from the demonstration, not a tail-latency study. Missing are depth-sensor failure rates, camera extrinsic drift, occlusion, simultaneous speakers, target disappearance, conflicting gesture requests, and long free-running drift. Public videos show that the system can operate in selected scenarios; they do not substitute for quantitative deployment tests.
Conclusion
XBot-Omni's contribution is an interface argument as much as a benchmark result. The system replaces text-and-waveform boundaries with native semantic and speech states, then adds explicit gesture events so that motion carries meaning and timing rather than only prosodic variation. The Thinker, Talker, and Actor form a coherent research architecture for coordinated speech, body movement, and facial expression.
The evidence shows real strengths: stronger audiovisual temporal understanding, substantial gains on context updating, high semantic gesture quality under the reported protocol, and large improvements in visually grounded pointing. It also exposes unresolved questions: absent public code, no unified end-to-end success metric, incomplete robustness analysis, and tradeoffs in pointing naturalness. XBot-Omni is therefore a strong systems hypothesis with promising component evidence, not yet a reproducible deployed solution.
Embodied communication is not a sequence of completed modules. It is a shared clock on which meaning, speech, and motion can condition one another while all three are still being generated.
