OpenBelief-Nav: Evidence-Preserving Object Memory for Open-Vocabulary Language-Guided Navigation

Paper metadata: Dinh Tuan Nguyen, Anh Dao, Phuong Nam Dang, Quan-Dung Pham, Tuyen P. Le, Truong Nguyen, and Quan Nguyen. Paper: https://arxiv.org/abs/2608.13923v1. Project page: https://openbelief-nav.github.io/.

Code status: No public implementation of the method was available when this analysis was written. The project page labels its code entry as coming soon, and the public repository found by searching is only the project website. The explanation below therefore comes from the paper's text, equations, figures, and tables rather than source-level reproduction. The authors state that code will be released after acceptance, but the current page should not be treated as a runnable system.

One-sentence summary: OpenBelief-Nav preserves competing open-vocabulary explanations for each object, reads that memory differently for semantic mapping and navigation, and verifies a reached candidate with RGB-D before rejecting it or confirming the target.

1. Motivation: language-guided navigation needs revisable memory

A language-guided robot is often described as a system that hears a target phrase, embeds it, and walks toward the best matching object. The harder problem begins after the first match is wrong. A room contains several visually similar objects, a detector changes its wording across viewpoints, and an early label may be supported by only one frame. A conventional semantic map can overwrite that label and leave the robot with no record of why an alternative was once plausible.

Open-vocabulary perception intensifies the issue. Vision-language models can propose names outside a fixed training taxonomy, but the same region may be called a monitor, a screen, a display, or a black device. Treating every phrase as a mutually exclusive class fragments the map. Collapsing every phrase too early creates the opposite failure: two similar objects become one node, or a mistaken label becomes the only explanation available to the navigator.

Scene-graph navigation systems commonly assign one category per node and retrieve with text similarity. They can work well for standard object classes, but they provide a weak interface for explaining a failed visit. OpenBelief-Nav treats semantic mapping as evidence accumulation rather than class assignment. Observations contribute support to concept clusters, and the robot chooses how to read the resulting distribution when a task arrives.

Figure 1 shows the complete pipeline. SAM 3 produces object masks, Qwen3-VL-4B-Instruct proposes open-vocabulary phrases, and MobileCLIP2-S4 maps visual crops and phrases into compatible feature spaces. Geometry-led association links observations across frames, synonym phrases are grouped into concept clusters, and Dirichlet-smoothed support forms a belief distribution. At navigation time, the same object memory can expose a consensus representation or a full belief readout.

Figure 1: Object memory, concept belief, and task-time readout form one navigation pipeline.

The central change is the map's semantic commitment. A node no longer claims that it has one correct class. It records which observations support which explanations. If a visit fails, the robot can exclude that candidate from the current query without erasing the object's language evidence. This is important in realistic homes, where two monitors or two cabinets may differ by context rather than by a single category word.

2. Object nodes, concept clusters, and task-time readout

The paper represents each object with position, visual features, observation history, vocabulary evidence, a concept distribution, a consensus text feature, and retrieval state. The node is written as:

$$ B_i=(P_i,f_i^v,O_i,V_i,p_i,f_i^t,\rho_i) $$

Here $P_i$ is the object position, $f_i^v$ is its visual feature, $O_i$ is the set of observations, $V_i$ is accumulated support over concept clusters, $p_i$ is the smoothed relative support distribution, $f_i^t$ summarizes the concepts in text space, and $\rho_i$ stores whether the object remains eligible for retrieval. Geometry, appearance, and language are therefore separate evidence channels rather than one overloaded category field.

A concept cluster is not a hand-written taxonomy. It emerges from phrases proposed by the vision-language model and grouped by semantic similarity. One cluster may contain monitor, screen, and display, while another may contain laptop. The goal is not to construct a perfect ontology. It is to avoid counting synonymous phrases as independent objects while retaining plausible distinctions that may matter after the robot approaches the candidate.

Task-time readout explains why the method should not be judged by one representation alone. A consensus text feature is compact and stable, while a full belief keeps low-probability alternatives available for detailed matching. The paper exposes both from the same memory instead of declaring that one must replace the other. This design choice becomes especially clear in the experiments, where the best semantic readout and the best navigation readout are not the same.

3. From observations to revisable beliefs

3.1 Instance masks, open-vocabulary phrases, and aligned features

The mapping input is an RGB-D sequence. SAM 3 separates object instances, Qwen3-VL-4B-Instruct proposes phrases for each region, and MobileCLIP2-S4 produces visual and text embeddings. Segmentation answers which pixels belong together, the language model offers possible descriptions, and the embedding model makes comparison possible. No single module is required to produce the final category on the first frame.

This separation matters for open-vocabulary operation. Multiple phrases may enter the object's concept set, while the visual feature and geometry remain a relatively stable identity anchor. Later observations can alter the language distribution without creating an entirely new map node. Open vocabulary is thus used not only to recognize more labels, but also to preserve more explanations that can later be tested.

3.2 Geometry-led association across frames

Two observations are associated only when their spatial relation and visual similarity both pass thresholds:

$$ o_e \sim o_{e'} \iff \Omega_r(P_e,P_{e'}) \geq \tau_g \ \text{and}\ \left\langle f_e^v,f_{e'}^v\right\rangle \geq \tau_v $$

$\Omega_r(P_e,P_{e'})$ measures spatial overlap, $\tau_g$ is its threshold, and the second condition compares visual features with an inner product. Geometry is checked first so that a model changing its wording does not split the same physical object into several nodes. The visual gate then prevents nearby but distinct objects from being merged merely because their positions are close.

3.3 Synonym consolidation and reliability-weighted support

An incoming phrase either adds support to an existing concept cluster or creates a new explanation. The observations are not weighted equally. Reliability is estimated as:

$$ w_e=c_e\sqrt{n_e} $$

$c_e$ is observation confidence and $n_e$ represents the amount of supporting evidence. The square root rewards additional evidence while limiting the dominance of one large detection batch. Accumulated support for concept $k$ on object $i$ is:

$$ V_{ik}=\sum_{e\in E_{ik}} w_e $$

The paper then converts support into a smoothed relative distribution:

$$ p_{ik}=\frac{V_{ik}+\alpha}{\sum_j V_{ij}+\alpha K_i} $$

$\alpha$ controls smoothing and $K_i$ is the number of concepts associated with the object. Smoothing prevents a single phrase from receiving probability one after one observation, and it leaves room for rare interpretations to be tested later. The distribution is a current allocation of evidence, not a claim that language categories have objective probabilities.

3.4 Consensus and full-belief semantic readouts

For efficient object retrieval, the system can merge concept embeddings into a normalized consensus feature:

$$ f_i^t=\operatorname{normalize}\left(\sum_k p_{ik}g_{ik}\right) $$

$g_{ik}$ is the text embedding of concept $k$. The resulting vector is stable and suitable for instructions such as finding a display in a room. It also discards some information: if two candidates share the dominant category, a small but relevant distinction may disappear from the aggregate.

The full-belief readout retains every concept contribution:

$$ H_i^{bel}(q)=\sum_k p_{ik}\left\langle g_{ik},f_q\right\rangle $$

The query feature $f_q$ is compared with each concept, and the support distribution determines how much each comparison matters. This readout preserves rare but useful explanations, which helps fine-grained semantic segmentation. It can also make a mistaken low-frequency phrase compete during navigation, explaining the task-dependent results reported by the authors.

3.5 Fusing language and visual appearance

Retrieval combines a visual score with the selected semantic readout:

$$ S_i^{(r)}(q)=\lambda_i\left\langle f_i^v,f_q^v\right\rangle +(1-\lambda_i)H_i^{(r)}(q) $$

$\lambda_i$ controls the balance, and $r$ identifies the current semantic readout. The selected object maximizes this score among candidates not already rejected for the query:

$$ \operatorname{retrieve}(q,R_q)=\arg\max_{i\notin R_q}S_i^{(r)}(q) $$

The exclusion set $R_q$ is query-specific. After a visit rejects an object, the set is updated as:

$$ R_q^{(k+1)}=R_q^{(k)}\cup\{i^{(k)}\} $$

The object remains in the map and can still be returned for a different instruction. This is a narrow but important distinction between query-level rejection and global deletion. A navigation failure changes what the robot has tested for this goal, not what it believes the entire scene contains.

3.6 Arrival verification and belief reweighting

Reaching a candidate creates new visual evidence. The system reweights the object's concepts using the current close-range feature:

$$ \tilde p_{ik}=\frac{p_{ik}^{\gamma}\exp\left(\left\langle g_{ik},f_i^v\right\rangle/T_{vis}\right)}{Z} $$

$\gamma$ sharpens the prior belief, $T_{vis}$ sets the visual temperature, and $Z$ normalizes the result. The updated belief is then projected onto candidate instances:

$$ \hat p_i(c)=(1-\lambda_{vis})\sum_k\tilde p_{ik}W_{ikc}+\lambda_{vis}v_i(c) $$

$W_{ikc}$ connects concepts to instances, $v_i(c)$ is the visual instance score, and $\lambda_{vis}$ balances the two sources. The robot can confirm the candidate, continue searching, or discard the candidate for this query. Figure 3 shows the real-world loop: the first desk location is rejected, the robot preserves the semantic object evidence, navigates to another candidate, and validates the correct bottle.

Figure 3: Real-world candidate rejection, re-retrieval, and arrival verification.

flowchart TD
    A[RGB-D observations] --> B[SAM 3 masks]
    A --> C[Qwen3-VL-4B phrases]
    B --> D[MobileCLIP2 visual features]
    C --> D
    D --> E[Geometry-led association]
    E --> F[Synonym concept clusters]
    F --> G[Reliability-weighted Dirichlet belief]
    G --> H[Object memory with evidence history]
    H --> I[Task-time readout]
    I --> J[Consensus retrieval]
    I --> K[Belief retrieval]
    J --> L[Navigate to candidate]
    K --> L
    L --> M[RGB-D arrival verification]
    M -->|confirmed| N[Return target]
    M -->|rejected| O[Exclude only for this query]
    O --> I

3.7 Deployment on a Unitree G1

The real robot is a Unitree G1 equipped with a RealSense D435i, a Livox MID-360, and an onboard Jetson AGX Orin. ROS 2 and Nav2 run on the robot for localization, planning, and motion. Semantic inference runs on an external workstation and communicates through Zenoh. This architecture reflects a practical split between embodied control and large-model inference: the language and vision models need not fit on the robot computer, but communication latency and service availability become part of the system boundary.

Figure 2: Unitree G1 platform, RGB-D camera, and LiDAR configuration.

3.8 What the system does not delegate to a language model

The language model does not directly output velocity commands. It proposes object concepts, the scene memory supplies spatial candidates, Nav2 generates a path, and close-range RGB-D evidence verifies the arrival. Each module has a defined evidence role. A failed visit is not silently converted into a new navigation policy, and a confident phrase is not treated as proof that the robot is standing beside the requested object.

This modularity is useful for robot developers. The semantic memory can be inspected and replayed, the navigation stack remains replaceable, and the vision-language service can be updated independently. The cost is a multi-service dependency chain. The paper does not report end-to-end service latency, network recovery, or failure rates for each interface, which limits operational conclusions.

4. Experiments: the best readout depends on the task

The evaluation uses ScanNet200, Replica, and HM3D-YCB. ScanNet200 stresses fine-grained open-vocabulary semantics, Replica provides cleaner indoor geometry, and HM3D-YCB contains 78 language-guided navigation trials. The physical pilot includes ten paired case studies and twenty runs on the Unitree G1. The paper's central empirical question is not whether one model wins every benchmark, but whether evidence-preserving memory can support different task-time reads.

For semantic segmentation, visual-only prediction gives mIoU 0.1286 on ScanNet200 and 0.2287 on Replica. The consensus representation raises these values to 0.2107 and 0.2831. Early commitment reaches 0.2393 on ScanNet200 but falls to 0.2701 on Replica. The full-belief readout is best on both datasets, at 0.2742 and 0.2912. Retaining multiple explanations therefore helps when the task requires detailed semantic distinctions.

Semantic readout	ScanNet200 mIoU	Replica mIoU	Interpretation
Visual only	0.1286	0.2287	No object-level language consensus
Consensus	0.2107	0.2831	Stable but compresses alternatives
Early commitment	0.2393	0.2701	Strong on ScanNet, weaker on Replica
Full belief	0.2742	0.2912	Best on both segmentation sets

Against an existing scene-graph method, full belief reaches 0.2742 on ScanNet200 versus 0.1604 for DualMap, and 0.2912 on Replica versus 0.2538. The comparison is not merely a larger segmentation backbone: the main change is how object nodes accumulate, represent, and read language evidence. The paper's representation-level ablation is therefore more informative than the headline number alone.

Method	ScanNet200 mIoU	Replica mIoU
DualMap	0.1604	0.2538
OpenBelief-Nav full belief	0.2742	0.2912
Absolute gain	0.1138	0.0374

Navigation results reverse the ranking. On HM3D-YCB, ConceptGraphs reaches 61.5% SR@1m, HOV-SG reaches 52.6%, and DualMap reaches 70.5%. The consensus and early-commit readouts of OpenBelief-Nav both reach 76.9%, while belief-weighted retrieval reaches 74.4%. A richer semantic distribution does not automatically produce better robot navigation; stable consensus can reduce competition from similar candidates.

Navigation method or readout	HM3D-YCB SR@1m	Observation
ConceptGraphs	61.5%	Comparison baseline
HOV-SG	52.6%	Below ConceptGraphs
DualMap	70.5%	Strong comparison
Consensus readout	76.9%	Tied best
Early commitment	76.9%	Tied best
Belief-weighted readout	74.4%	Retains alternatives but slightly lower

The ablation supports the default task interface. With consensus as the default, ScanNet200 mIoU is 0.2107, navigation is 76.9%, and YCB target success is 61.1%. Removing low-evidence regularization leaves ScanNet almost unchanged at 0.2095 but reduces navigation to 70.5% and YCB target success to 50.0%. Equal observation weighting gives 0.2010, 70.5%, and 55.6%, respectively. Low-support phrases should not outvote repeatedly observed concepts merely because they are numerous.

Consensus ablation	ScanNet200 mIoU	Navigation SR@1m	YCB target success
Default	0.2107	76.9%	61.1%
No low-evidence regularization	0.2095	70.5%	50.0%
Uniform observation weights	0.2010	70.5%	55.6%

Figure 4: Qualitative comparison of object-level semantic readouts.

In the physical pilot, the system achieves 6/10 top-1 success and 8/10 full recovery after one or more corrections. Under confusable conditions, top-1 success is 1/5 and full recovery reaches 3/5. Full recovery visits 1.40 candidates on average, or 1.80 under confusable conditions. These numbers support feasibility of the recovery loop, but ten paired cases are not enough to estimate broad deployment reliability.

Figure 4 clarifies the tradeoff behind the tables. A full belief can help segmentation by preserving a small but relevant interpretation, while a navigation query may benefit from a more stable consensus. The paper therefore does not advertise a universally superior representation. It exposes task-time readout as an explicit interface between semantic memory and embodied goals.

5. Limitations: recovery works, but only inside a narrow boundary

The authors explicitly limit correction to targets that already exist in the map. If an object was never observed, has been removed, or is missing because of a structural mapping error, the system does not infer it from the language query. General map repair is outside scope. Open-vocabulary language therefore does not imply complete open-world perception; unobserved space remains a hard boundary.

The physical evaluation is a ten-case pilot. It does not establish generalization across homes, robot platforms, or broad instruction distributions, and twenty runs cannot estimate long-tail failure rates. Deformable objects, moving people, and dynamic occlusion can invalidate a frozen map. The correction mechanism relies on an existing candidate rather than continuously rebuilding the environment.

The method also assumes that the navigation stack can reach the candidate viewpoint. If a door is locked, a path is blocked, or motion fails, the robot cannot count that failure as semantic negative evidence. Separating semantic rejection from navigation failure is necessary, but a complete recovery policy is not demonstrated. A reported navigation success rate should not be mistaken for a pure measure of semantic reliability.

An independent limitation is the frozen-map assumption. The experiments emphasize retrieval and verification from accumulated memory, not continuous map evolution. A long-running robot must decide when an observation becomes stale, whether an object moved, and when to re-segment. Keeping all evidence forever produces outdated support; forgetting too aggressively recreates the label-overwrite problem that the paper tries to avoid.

Finally, the method has no public implementation and reports limited runtime decomposition. Semantic inference runs on an external workstation through Zenoh, making the system sensitive to network and compute configuration. The paper does not provide end-to-end retrieval latency, verification time, service interruption behavior, or model compute budgets. Those measurements are necessary for reproduction and deployment planning.

6. Conclusion

OpenBelief-Nav changes an open-vocabulary object map from a table of labels into a distribution of evidence. Geometry-led association stabilizes object identity, Dirichlet smoothing preserves uncertain explanations, and task-time readout lets one memory serve semantic and navigation goals. Query-specific rejection allows the robot to recover after visiting the wrong instance without deleting the object globally.

The experiments show that the most important result is task dependence. Full belief is best for ScanNet200 and Replica segmentation, while consensus and early commitment reach 76.9% on HM3D-YCB navigation. This supports decoupling scene representation from readout policy. Dynamic map maintenance, explicit navigation-failure handling, cross-environment evaluation, and public runtime measurements are the next questions raised by the work.

Golden line: A revisable map is not a collection of always-correct labels; it is a memory that lets the robot admit a wrong turn without losing the evidence needed to choose again.