Holi-World: Building Holistic World Intelligence from Human-Ego Experience

Paper metadata

Paper: Holi-World: Building Holistic World Intelligence from Human-Ego Experience.

Authors: Hao Li, Xiyuan Zheng, Yuanyuan Gao, Yalun Dai, Xinhao Ji, Xuan-Li Chen, Jing Zhao, Fangzhou Hong, Zhaoxi Chen, and Ziwei Liu.

Affiliations: Ropedia, Nanyang Technological University, HKUST, and Peking University.

Links: arXiv:2610.26549 and the project page.

Release status: The project repository and demonstration are public, but its README states that the code will not be open-sourced. The model and dataset are marked as coming soon. This analysis is based on the complete PDF, LaTeX source, project page, and public repository; it does not treat unavailable assets as released.

One-sentence summary

Holi-World converts human egocentric walking and manipulation recordings into traceable world records with persistent 3D structure, object identities, motion trajectories, and supporting evidence, then uses those records to train a vision-language model for spatial and temporal reasoning.

Technical background and scope

The world state in Holi-World is not a full physical simulator. At time $t$, it combines input observations, 2D detections and masks, a Gaussian scene, a point cloud, persistent static boxes, time-indexed dynamic boxes, and semantic evidence. Static instances remain linked across observations, while dynamic instances retain their states at each time step.

Apparent motion in egocentric video has two sources: camera motion and object motion. A system that treats 2D displacement as object motion will mistake a head turn for a moving chair. HoliWorld lifts pixels into world coordinates using depth, intrinsics, and camera poses, then fits and links boxes in that shared frame.

The dynamic pipeline explicitly assumes at most one moving object at a time. This narrows the identity association and mask propagation problem enough to test long interruptions and out-of-view returns. It does not cover two-handed manipulation, multiple people exchanging objects, or several independently moving entities. That restriction is stated in the paper and is one of the method's concrete boundaries.

The components are diverse, but each has a clear evidence role. Stereo RGB and IMU establish motion. Depth Anything 3 and Gaussian reconstruction establish metric geometry. Open-vocabulary detection and point-cloud detection propose objects. SAM supplies masks. A vision-language model routes objects, describes interactions, and audits static candidates. The scientific question is how these roles fit together, not whether one module wins a standalone benchmark.

Method: constructing a world record from the camera stream

1. Shared geometry before the static and dynamic split

HoliWorld begins with visual-inertial SLAM over synchronized stereo RGB and IMU data. It uses the recovered poses and RGB keyframes with Depth Anything 3 to estimate depth and dense points, optimizes a 3D Gaussian scene, renders refined depth at observed poses, and completes dynamic regions with per-frame depth estimates. The retained assets are the Gaussian scene, the reconstructed point cloud, and time-aligned observations.

The observation set is the paper's first numbered equation. It binds each image to its depth, intrinsic matrix, and camera-to-world transform.

$$\mathcal{O}=\{(I_t,D_t,\mathbf{K}_t,\mathbf{T}_t)\}_{t=1}^{T}$$

Here $I_t$ is the RGB frame, $D_t$ is the depth map, $\mathbf{K}_t$ is the intrinsic matrix, and $\mathbf{T}_t$ maps camera coordinates into the shared world frame. A vision-language model extracts categories, attributes, motion cues, and interaction descriptions from keyframes. Categories steer static proposals; motion and interaction cues identify dynamic targets.

This shared preparation is the equivalent of a common ledger. If the static and dynamic branches estimated separate coordinate systems, two plausible boxes could still refer to different places in the same room. Shared poses, depth, and points make later fusion an actual merge rather than a concatenation of incompatible outputs.

2. Static perception as a dual-path proposal system

One static path runs Boxer separately on the left and right RGB streams. Open-vocabulary 2D detections are lifted into metric 3D candidates using depth and camera poses, then associated across frames. This path benefits from image detail and small-object evidence, but one view may expose only a single side of the object.

The second path runs TR3D on the reconstructed scene point cloud. It captures large extents accumulated across viewpoints and is particularly useful for beds, cabinets, and tables. Its weakness is the opposite one: objects with weak point support may be missed. Instead of forcing one detector to dominate, the system preserves proposals from both.

The paper describes the image-lifting process in prose. The following expression makes that description executable; it is not presented as a separate numbered equation in the paper:

$$\mathbf{B}^{I}_{j,t}=F_{\mathrm{box}}\left(\mathbf{K}_t,\mathbf{T}_t,D_t,r^{I}_{j,t}\right)$$

In this notation, $r^{I}_{j,t}$ is a 2D detection region and $F_{\mathrm{box}}$ denotes lifting plus 3D box fitting. Multiple proposals may refer to the same object, so low-cost geometric checks merge spatially compatible candidates and discard boxes with too little point support. The system also keeps one fallback for a selected static box: the rank-1 alternative from the same candidate cluster, or a directly suppressed candidate linked by IoU suppression.

A vision-language model then inspects each selected box against its source observations and supplementary views rendered from the Gaussian scene. It may keep the geometry, refine it, change the label, or reject the candidate. A rejected primary triggers at most one evaluation of its fallback, using that fallback's own evidence. Accepted boxes form $\mathcal{B}^{\mathrm{sta}}$ and retain links to supporting observations.

The division of labor is deliberately conservative. Detectors create candidates, geometry removes obvious unsupported boxes, and the vision-language model performs semantic review on an evidence packet rather than hallucinating boxes from text alone. The single-fallback rule limits runaway correction and keeps the output tied to a concrete candidate.

3. Dynamic identity continuity across interaction segments

A manipulated object can disappear into a drawer, leave the camera view, and return later. Frame-by-frame detection would assign a new identity on each return. HoliWorld instead uses hand regions and VLM interaction descriptions to identify the target within each segment, then compares descriptions and keyframe evidence across segments. This temporal label continuity becomes the anchor for mask selection and tracking.

SAM initializes and propagates a target mask within each interaction segment. The paper denotes the foreground mask at time $t$ as $m_t$. For every valid masked pixel, the system uses the completed depth map and camera pose to recover object-supported world points. The backprojection is the paper's second numbered equation.

$$\mathcal{X}_t=\left\{\mathbf{T}_t\left(D_t(\mathbf{u})\mathbf{K}_t^{-1}\bar{\mathbf{u}}\right)\mid\mathbf{u}\in m_t,\;D_t(\mathbf{u})\ \text{is valid}\right\}$$

$\bar{\mathbf{u}}$ is a homogeneous image coordinate, and $\mathbf{T}_t(\cdot)$ applies the camera-to-world rigid transform. The set $\mathcal{X}_t$ contains only world points supported by the tracked mask and valid depth. Unsupported samples are filtered further before fitting an oriented box. A set-level expression for that filtering step is:

$$\mathcal{X}^{\star}_t=\{\mathbf{x}\in\mathcal{X}_t\mid \mathbf{x}\ \text{passes depth and support filters}\}$$

The paper does not prescribe one closed-form oriented-box solver, but the resulting state can be represented by a center, three extents, and a rotation. This notation restates the stated output form rather than claiming a new equation from the authors:

$$\mathcal{B}^{\mathrm{dyn}}_{t,k}=(\mathbf{c}_{t,k},\mathbf{s}_{t,k},\mathbf{R}_{t,k})$$

Boxes with the same identity are linked across time into a 4D track. The essential asset is not a perfect mask in one frame. It is the agreement among object identity, image-space mask, world-space points, and box state over the interaction.

The evaluation includes periods when the target is out of view. Missing predictions remain failures, so success requires memory and identity continuity rather than scoring only the frames where the object is easy to see. Figure 5 shows predicted trajectories as solid curves and reference trajectories as dashed curves.

4. A unified, time-indexed state

The static boxes and dynamic tracks meet in the world state of the paper's third numbered equation:

$$\mathcal{S}(t)=\left(\mathcal{O}_t,\mathcal{U}(t),\mathcal{G},\mathcal{P},\mathcal{B}^{\mathrm{sta}},\mathcal{B}^{\mathrm{dyn}}(t),\mathcal{Z}(t)\right)$$

$\mathcal{U}(t)$ contains detected 2D boxes and tracked masks; $\mathcal{G}$ and $\mathcal{P}$ hold the Gaussian scene and point cloud; $\mathcal{B}^{\mathrm{sta}}$ contains persistent static boxes; $\mathcal{B}^{\mathrm{dyn}}(t)$ contains dynamic states at that time; and $\mathcal{Z}(t)$ stores labels, identities, confidence, interaction semantics, and evidence links.

Static instances persist across observations, while dynamic instances have explicit temporal states. This lets a question distinguish between an object that remained in the room and an object that moved from a table into a hand. Every record can be traced to an object, time, and supporting observation, which is the mechanism that prevents the QA layer from becoming detached from perception.

5. Turning world records into grounded supervision

The source data contain roughly 100 scenes, 444 egocentric videos, about 100 Gaussian reconstructions, 1.322 million RGB frames with depth and camera poses, 90K 2D boxes, 2,466 static 3D boxes, and 2,095 dynamic tracks. These records support about 4K 3D grounding annotations and 80K captions. HoliWorld-Data organizes them into raw observations, reconstructed geometry, object and interaction semantics, and spatial-intelligence supervision. Spatial QA derives direction, distance, containment, support, and viewpoint-dependent relations from object geometry and camera poses. Temporal QA compares the same identity over time to describe movement, duration, and event order. Interaction QA uses manipulation evidence and state changes.

The dataset defines 30 fine-grained QA types. Eleven static types cover camera motion and object-level relations across single and multiple views. Seven trajectory types cover motion status, timing, direction, sequence, and path properties. Twelve caption-based types cover action recognition, object interaction, temporal grounding, interaction phases, event order, state transitions, and contrastive hard negatives.

There are 13,647 static QA pairs and 64,409 dynamic QA pairs. The distinction matters: the former tests geometric relations at a selected time, while the latter tests identity-linked state changes across time. The roughly 80K total is therefore heterogeneous supervision rather than one question template repeated at scale.

Answers include categorical labels, numerical values, multiple-choice options, time intervals, and free-form descriptions. The paper states that each example retains links to source observations and annotations. A compact evidence tuple can therefore be written as follows; this is an organizational restatement of that stated requirement, not a numbered formula from the paper:

$$e(q,a)\leftarrow(o_{t_1},o_{t_2},b_{t_1},b_{t_2},c_{t_1:t_2})$$

Here $o_t$ denotes observations, $b_t$ denotes object states or tracks, and $c$ denotes aligned caption or interaction evidence. The tuple is not meant to encode every QA type. It expresses the shared property that a target answer is connected to the perceptual records that justify it.

flowchart LR
    A[Ego RGB and IMU] --> B[VI-SLAM and DA3]
    B --> C[3DGS and Point Cloud]
    B --> D[VLM Semantic Cues]
    C --> E[Image and Point Cloud Candidates]
    D --> E
    E --> F[Geometric Merge and Filter]
    F --> G[VLM Validation and Fallback]
    G --> H[Static 3D Boxes]
    D --> I[Interaction Segments]
    I --> J[Cross-Segment Identity Association]
    J --> K[SAM Mask Tracking]
    K --> L[World Point Backprojection]
    L --> M[Oriented Boxes and 4D Tracks]
    H --> N[Unified World State]
    M --> N
    N --> O[Grounded Captions and QA]
    O --> P[HoliWorld-VL]
    N --> Q[Human-Annotated Bench]
    P --> Q

The diagram follows the actual dependencies in the method. Shared geometry and semantic cues route objects into complementary branches. Both branches produce records in one world state, which then supports supervision and independent evaluation.

Static 3D perception results

Static evaluation uses 20 sequences each from ScanNet and ScanNet++, plus 20 manually annotated in-house sequences. AP is measured at 3D IoU thresholds of 0.25 and 0.50. Matched-box mIoU is averaged over pairs matched at IoU 0.25. All scores use a zero-to-one scale.

HoliWorld leads both AP thresholds on all three datasets. On ScanNet it reaches 0.68 AP25 and 0.55 AP50, compared with 0.40 and 0.12 for the strongest baseline, Map-Det3D. The AP50 increase is especially informative because strict IoU matching leaves less room for loosely placed boxes.

The lower absolute scores on ScanNet++ and the in-house benchmark expose how much remains difficult. AP25 falls to 0.30 and 0.24, while AP50 falls to 0.13 and 0.12. A leading result therefore does not mean the scene has been solved. It means the combined candidate and validation strategy handles these harder captures better than the tested alternatives.

The appendix notes an asymmetry between AP and matched-box mIoU. On the in-house data, the AP gain over Boxer is larger than the mIoU gain among successfully matched boxes. The primary improvement is better object coverage, while precise extent fitting remains a secondary challenge. Since mIoU is conditional on matching, it should not be interpreted as accuracy over all objects.

Dynamic tracking and trajectory recovery

Dynamic evaluation uses 20 sequences each from Aria Digital Twin and HoliWorld-Bench. It covers the full RGB timeline, including periods when objects are out of view. HOTA, association accuracy, and detection accuracy use a 0.1 meter world-center tolerance. PCL counts correctly localized reference instance-frames under a sequence-global assignment; missing trajectories and localization failures count as incorrect.

Several baselines receive ground-truth first-visible masks or category queries, while HoliWorld uses no annotation-derived test-time input. The comparison therefore tests whether a self-contained pipeline can recover identity and trajectory from its own observations.

On ADT20, HOTA rises to 54.31 percent, 18.31 points above IT3DEgo with ground-truth-guided initialization. Association accuracy reaches 71.94 percent, 18.34 points above the same baseline. Because HOTA jointly reflects detection and association, the gap cannot be explained by reproducing a good mask on visible frames alone.

The internal benchmark shows a larger separation. HoliWorld reaches 43.14 percent HOTA versus 15.13 percent for SAM+LMK, and PCL rises from 23.68 to 57.89 percent. The in-house scenes include longer interaction gaps and out-of-view intervals, where frame-level detections lose the object and track continuity becomes necessary.

The full table also prevents an overly clean narrative. On HoliWorld-Bench, HoliWorld's TP-L2 error is 0.0280 meters, not the best value in the comparison. Its advantage is primarily recall, coverage, and identity association. A useful next step is to separate identity recovery from metric refinement, because a system can keep the correct track alive while still placing its center less accurately than a specialist.

Does generated supervision improve reasoning?

The authors fine-tune Qwen3.5-4B on HoliWorld-Data to obtain HoliWorld-VL-4B. They report Token F1, yes/no accuracy, multiple-choice accuracy, and numerical accuracy on HoliWorld-Bench and EgoDynamic4D. EgoDynamic4D uses a different source of dynamic scenes, so its results provide some evidence of transfer beyond the internal benchmark.

Token F1 increases from 16.35 to 78.02 percent on HoliWorld-Bench and from 18.29 to 66.39 percent on EgoDynamic4D. The 4B fine-tuned model also beats the zero-shot 27B model on all eight reported metrics. This supports the argument that task-aligned, evidence-grounded supervision can be more valuable than a larger general-purpose model without that alignment.

The gains are not uniform across question formats. On the internal benchmark, numerical accuracy reaches 81.84 percent, but on EgoDynamic4D it is only 16.33 percent. Yes/no accuracy is 99.67 percent on the latter. A language model can answer many binary questions from strong object and interaction priors while still failing when it must estimate a time interval, distance, or continuous state.

That gap is useful for interpreting the data recipe. Training on static geometry, dynamic tracks, and captions improves several formats, but EgoDynamic4D's near-zero-to-sixteen percent numerical change shows that open numerical reasoning remains difficult. More data or a larger model alone may not solve it unless the model learns to select the exact evidence and perform the required geometric or temporal computation.

Ablation: complementary evidence matters more than any single module

The static ablation shows why two detection paths are retained. Removing Boxer's image-lifting path lowers ScanNet AP25 from 0.68 to 0.58 and the in-house score from 0.24 to 0.10. Removing TR3D lowers ScanNet AP25 to 0.32 and the in-house score to 0.20. Neither branch substitutes for the other across object scales and viewpoints.

Caption guidance also matters more on the harder in-house set: removing it gives 0.21 AP25 versus 0.24 with the full system. VLM verification barely changes geometric scores, with ScanNet mIoU unchanged at 0.72 and the in-house mIoU moving from 0.52 to 0.53. The qualitative value is mainly label correction and rejection of unsuitable candidates.

The dynamic ablation produces a clearer interaction effect. With SAM association but no label continuity, HOTA is 44.09 percent on ADT20 and 32.86 percent on the in-house set. With label continuity but no SAM association, HOTA is 38.38 and 38.89 percent. The complete configuration reaches 54.31 and 43.14 percent.

The in-house dynamic ablations cross over on different metrics. Without label continuity, HOTA is lower but PCL is 45.30 percent. Without SAM association, HOTA is higher but PCL is 51.04 percent. The full model reaches 43.14 percent HOTA and 57.89 percent PCL. Label continuity answers whether the target is the same instance, while SAM association preserves its image-space evidence over time.

Limitations

The first limitation is stated directly in the paper: Foundry assumes at most one moving object at each time. Real homes and warehouses contain bimanual manipulation, multiple people exchanging tools, and independent background motion. Extending the method requires new rules for target assignment, overlapping masks, crossing trajectories, and concurrent temporal evidence.

The second is error propagation. Static output depends on detection, geometric merging, filtering, fallback selection, and VLM review. Dynamic output depends on interaction segmentation, identity association, SAM propagation, valid depth, point filtering, and box fitting. A failure in any stage can enter the training set, and the trained model may learn the generator's systematic biases. The human benchmark has only 20 in-house sequences.

The third is reproducibility. The model and dataset are described as coming soon, and the repository says the core code will not be released. The paper gives method details, evaluation protocols, baseline adaptations, and matching rules, but external researchers cannot yet run the exact Foundry pipeline or regenerate the same records. This limits independent validation even though the architecture is described.

The fourth is the model comparison boundary. The downstream study centers on fine-tuning Qwen3.5-4B and compares it with a zero-shot Qwen3.8-27B model. It does not establish that the same supervision has equal value for other vision-language architectures, longer video context, or additional sensors. The 16.33 percent numerical score on EgoDynamic4D is a concrete sign that continuous reasoning remains unsolved.

Finally, egocentric records contain private homes, faces, personal objects, and detailed hand activity. The appendix explicitly requires consent, review, redaction, and compliance with dataset and model licenses. A world record makes those observations easier to link and search, so release controls must address privacy and misuse in addition to benchmark accuracy.

Conclusion

Holi-World contributes an end-to-end data loop rather than a standalone detector. Foundry constructs persistent static boxes and time-indexed dynamic tracks in a shared coordinate system. Data converts those records into grounding, captions, and spatial-temporal QA with evidence links. VL tests whether the supervision teaches reasoning, while Bench uses independent human annotation to evaluate both the records and the model.

The empirical story is consistent across stages. Static perception leads on three datasets, dynamic tracking leads on ADT20 and the in-house benchmark, and a 4B model trained on HoliWorld-Data outperforms a zero-shot 27B model across the reported question formats. The transferable design principle is the combination of identity continuity, world coordinates, and traceable evidence.

The next technical steps are multi-object and bimanual tracking, articulated and deformable states, longer temporal evidence selection, and calibrated uncertainty that can trigger targeted human review. If the planned model and dataset are released, independent evaluations should test not only average scores but also how often automatic errors survive into training and where numerical or long-horizon reasoning breaks.

Golden line

World intelligence is not built by feeding more video to a model. It is built by preserving the same identity, the same coordinates, and the same evidence chain as an object crosses view, time, and interaction.