Room-Mediated Co-occurrence for Zero-Shot Object-Centric Semantic Navigation via Frontier Scoring

Paper information: Adam Scicluna, Gavin Paul, and Alen Alempijevic, Robotics Institute, University of Technology Sydney, IROS 2026.

Paper and code: arXiv:2607.25448; project page; UTS-RI/RPV-SemNav. The official implementation is public and builds on the VLFM mapping and navigation stack in Habitat.

One-Sentence Summary

The paper turns object-target relevance into an overlap between room-probability distributions, propagates that evidence through navigable free space, and uses geodesic frontier scores to reach 0.539 success on the HM3D validation split without task-specific policy training.

Research Context and Motivation

Object-goal navigation asks a robot to find a language-named object in an unfamiliar indoor scene. The robot must solve two coupled problems at the same time: decide which floor space is traversable, and decide which unknown region is worth visiting first. A metric map can answer the first question, but a search strategy needs semantic knowledge about how rooms and objects are normally arranged. A bed is a useful cue for a bedroom, while a workbench is more likely to imply a garage or workshop.

Learning-based ObjectNav systems often acquire this knowledge from large annotated datasets, demonstrations, or reinforcement-learning environments. They can become strong within a fixed ontology, but a new target label may require additional supervision. End-to-end policies face a similar constraint: the training distribution largely determines what spatial relations the policy absorbs. These methods remain important performance references, but they do not directly provide a training-free route to a new vocabulary.

Zero-shot methods replace closed-set learning with vision-language models. A common pipeline compares a target string against an egocentric image embedding and projects the resulting similarity into a value map. This is flexible and relatively simple, but the semantic unit is the whole image. The score can therefore mix the target, background walls, furniture, lighting, and camera framing. It also depends on handcrafted prompts and on what happens to be visible in the current view.

Object-centric alternatives treat detected objects as spatial anchors. A segmented chair, sink, or desk can provide a more local signal than a whole-frame embedding. The difficulty moves to the relation between labels: raw CLIP similarity measures semantic proximity in language, not the probability that two object categories occur in the same physical room. Linguistically related objects can occupy separate functional zones, while weakly similar objects can be strong architectural cues.

The authors introduce room type as an interpretable intermediate variable. Each object label is first mapped to a probability distribution over nine room categories. Two objects are then compared by the overlap of those distributions. This replaces a high-dimensional, difficult-to-interpret object-object comparison with a small probabilistic space tied to indoor layout. The remaining problem is to prevent semantic evidence from crossing walls, which motivates propagation over free-space geometry.

Task Definition and Control Interface

An episode starts from an arbitrary pose in an unknown scene. At each step the agent receives an RGB-D observation and selects one of four actions: move forward 0.25 meters, turn left 30 degrees, turn right 30 degrees, or stop. The first twelve actions are left turns, giving the robot a complete panoramic survey before exploration begins. Once a target instance is detected, the policy switches from frontier search to approach behavior.

Success requires stopping within one meter of any correct target instance within 500 steps. The main metrics are Success Rate, Success weighted by inverse Path Length, and Distance to Goal. SR measures whether the episode ends successfully, SPL discounts success when the route is inefficient, and DTG records the remaining spatial error when the agent stops. The three metrics prevent a system from looking good only because it reaches goals through excessively long trajectories.

All compared methods use the same pretrained point-navigation controller. The experimental comparison therefore isolates high-level semantic frontier selection more cleanly than a setup in which every paper also changes low-level locomotion. This design choice matters because a stronger collision-avoidance policy could otherwise create an apparent gain that has little to do with room-mediated reasoning.

Method Overview

The pipeline begins with structural masking and open-vocabulary perception. It proposes object labels, refines them into instance masks, and combines mask centroids with depth to place detections in the world frame. Both the detected object labels and the target label are encoded as room-probability vectors. Their dot product becomes a co-occurrence amplitude. That amplitude is propagated from object positions through the occupancy map with the Fast Marching Method, and frontiers are ranked by the median propagated value in their local neighborhoods.

Figure 1 (paper Figure 1): Graphical overview. While searching for a microwave, visible objects that are more likely to share a room with the target emit stronger signals, while unrelated objects do not dominate the map.

flowchart TD
  A[RGB-D observation] --> B[Mask2Former ADE20K masking]
  B --> C[YOLOE proposals over 1203 LVIS labels]
  C --> D[SAM 3 masks and centroids]
  D --> E[Tracked labels and world positions]
  E --> F[CLIP text encoder]
  G[Target label] --> F
  H[Nine room names] --> F
  F --> I[Room probability vectors]
  I --> J[RPV dot product]
  J --> K[Square-root weight and adaptive sigma]
  L[Occupancy grid] --> M[FMM geodesic propagation]
  K --> M
  M --> N[Semantic value map]
  N --> O[Median value near each frontier]
  O --> P[Highest-scoring frontier]
  P --> Q[Shared PointNav controller]

Workflow 1: A paper-specific diagram drawn from the method. The distinctive path is room-probability projection, distribution overlap, geodesic propagation, and frontier median scoring.

Open-Vocabulary Perception

The perception stack first runs Mask2Former trained on ADE20K and blurs structural regions such as floors, walls, ceilings, windows, and curtains. These areas occupy large image regions but provide little evidence about which room contains a portable target. Removing their texture reduces spurious detections before the open-vocabulary detector sees the frame. The masking stage is therefore a semantic filter, not a replacement for object detection.

YOLOE then proposes objects over 1,203 LVIS labels. Detections with confidence above 0.5 are retained. The broad vocabulary is necessary for open-set navigation, but it also creates a precision problem: a proposal may be semantically plausible while its box is poorly localized. SAM 3 receives each detected label as a prompt and produces an instance mask. The mask is paired with depth to estimate a world-space centroid.

The target label is always included in the SAM 3 prompt list. This means the robot does not need a perfect LVIS proposal for the requested category before it can recognize a target. Once a target instance is confirmed, the policy can leave semantic exploration and approach the detected object. The detector-proposer and segmentation-refiner split supports open vocabulary while keeping the map signal attached to a localized physical region.

Full perception runs every three exploration steps, while navigation runs SAM 3 every three steps without repeating the complete detection stack. The authors justify this throttling by the limited change between consecutive RGB-D frames under a discrete action space. The schedule reduces computation, but it also means the reported runtime is not a per-frame worst case for every model in the cascade.

Figure 2 (paper Figure 3): The three-stage perception pipeline. Structural regions are masked, an open-vocabulary detector proposes labels, and prompted segmentation produces instance masks.

Room Probability Vectors

The room lexicon contains bathroom, bedroom, dining room, garage, kitchen, hallway, laundry, living room, and home office. Its role is not to classify the robot's present room. It supplies a shared coordinate system in which every object label can be described. Because the vocabulary is compact, the mapping is cheap to compute and easy to inspect. The authors also argue that the categories can be replaced by functional zones or domain-specific spaces without retraining the navigation policy.

For an object label $o$, the text encoder produces an embedding $e(o)$. The similarity to room label $r_i$ is measured by normalized cosine similarity:

$$s_o(r_i)=\cos\left(e(o),e(r_i)\right)=\frac{e(o)\cdot e(r_i)}{\lVert e(o)\rVert\lVert e(r_i)\rVert}.$$

The cosine value is transformed into a categorical distribution with temperature-scaled softmax:

$$p_o(r_i)=\frac{\exp\left(s_o(r_i)/\tau\right)}{\sum_{j=1}^{K}\exp\left(s_o(r_j)/\tau\right)}.$$

Here, $K=9$ and the paper uses $\tau=0.03$. The low temperature makes clear room associations sharper, which helps objects such as a bed or bathtub. It also amplifies noise in CLIP's text geometry, so an impulsive temperature choice can create overconfident room assignments. The resulting vector is $\mathbf{p}_o=[p_o(r_1),\ldots,p_o(r_K)]^\top$, a nine-dimensional representation independent of the robot's current pose and the visual appearance of a particular instance.

The source implements the same sequence in vlfm/vlm/clip.py:63-109: cosine similarity, temperature-controlled softmax, and matrix dot products. The room names are fixed in vlfm/vlm/room_types.py:1-11. This mapping is analytical. No relation classifier is trained on object-room pairs.

Room-Mediated Co-occurrence

Let $o_d$ be a detected object and $o_t$ the target. Their co-occurrence amplitude is the dot product of the two room-probability vectors:

$$A_o=\operatorname{cooc}(o_d,o_t)=\mathbf{p}_{o_d}\cdot\mathbf{p}_{o_t}=\sum_{i=1}^{K}p_{o_d}(r_i)p_{o_t}(r_i).$$

Each term is the probability mass the two labels assign to the same room. If both peak in the kitchen, the score is high. If one concentrates in a bathroom and the other in a garage, the product suppresses the mismatch. The calculation is symmetric, while the data-defined conditional probability $P(t\mid o)$ is not. The authors acknowledge this mismatch and test the choice empirically instead of claiming formal equivalence.

The implementation first computes a room distribution for the target, then multiplies a batch of detected-object distributions by it. Only positive instances are projected into world coordinates and added to the value map. The policy-side flow appears in vlfm/policy/itm_policy.py:252-310, while target and detection probabilities are produced in vlfm/vlm/clip.py:164-184.

The dot product is useful because the navigation task mainly needs to surface a few strong anchors. It multiplies probabilities, so a room supported by only one distribution contributes little. This sharpens peak agreement but compresses weaker associations. The later square-root transform is designed to expand part of that compressed low-score range without abandoning the multiplicative behavior that performs well in the top-five ranking analysis.

Figure 3 (paper Figure 2): System overview. Detected object labels are grounded as spatial signals, combined through room-probability overlap, and used to rank frontiers.

Geodesic Signal Propagation

Every accepted object emits a semantic signal into the map. The signal decays with distance, but distance is not Euclidean. The system runs the Fast Marching Method through the free-space occupancy grid, so propagation follows corridors, passes through doorways, and bends around obstacles. A wall blocks the signal from bleeding into an adjacent but disconnected room.

At a free-space cell $\mathbf{x}$, the value aggregates all object signals that can reach it. The set $\mathcal{O}_{\mathbf{x}}$ contains those reachable objects, $d_{\mathrm{geo}}$ is geodesic distance, and $\sigma_o$ controls spatial spread:

$$V(\mathbf{x})=\frac{1}{|\mathcal{O}_{\mathbf{x}}|}\sum_{o\in\mathcal{O}_{\mathbf{x}}}A_o\exp\left(-\frac{d_{\mathrm{geo}}(\mathbf{x},\mathbf{x}_o)^2}{2\sigma_o^2}\right).$$

The divisor prevents a cluttered area from winning simply because it contains many weak detections. After normalization, the value is closer to the representative relevance of the anchors available at that cell. Without it, a room full of ordinary furniture could overpower a nearby high-specificity cue such as a bathtub or bed.

The spread is not fixed. The method first computes a bounded weight from the co-occurrence amplitude:

$$w=\sqrt{A_o}.$$

It then maps that weight into the interval between minimum and maximum diffusion scales:

$$\sigma_o=\sigma_{\min}+(\sigma_{\max}-\sigma_{\min})w,$$

with $\sigma_{\min}=1.0$ meter and $\sigma_{\max}=5.0$ meters. Strong cues influence roughly room-scale areas, while weak cues remain local. The square root helps distinguish co-occurrence scores concentrated in the lower range. The corresponding implementation is vlfm/mapping/value_map.py:556-566; the FMM propagation and exponential decay are in the surrounding value-map code around lines 300-410.

The source uses scikit-fmm to propagate travel time through the navigable grid. If an object centroid lies on an obstacle, the implementation searches a nearby free-space seed and applies a morphological boundary rule to avoid a one-sided bias. Failed propagation falls back to an isotropic Gaussian. That fallback is practical, but the paper does not report how often it occurs, so the reported result cannot be read as proof that every update preserves free-space topology.

Figure 4: Qualitative search sequence for a bed. Semantic evidence accumulates around the bedroom region and reshapes the frontier choice as exploration proceeds.

Frontier Scoring

The system extracts boundaries between known free space and unknown space, clusters nearby boundary cells, and treats each cluster as a frontier candidate. For frontier $F_j$, the score is the median value over a circular neighborhood $\mathcal{C}_j$:

$$V_j=\operatorname{median}_{c\in\mathcal{C}_j}V(c).$$

A median is less sensitive than a maximum to one isolated hot cell. Zero-valued and unobserved cells are removed from the neighborhood; if no nonzero evidence remains, the function returns a negative value so that an empty frontier is not mistaken for a strong candidate. The final selected frontier is:

$$F^{*}=\arg\max_{F_j\in\mathcal{F}}V_j.$$

The neighborhood reduction is implemented in vlfm/utils/img_utils.py:213-264, and the frontier sorting path is in vlfm/mapping/value_map.py:734-775. Frontier extraction itself lives in frontier_exploration_src/frontier_exploration/frontier_detection.py:15-61, with nearest-waypoint selection around lines 225-242.

This scoring strategy exploits semantic evidence but does not add an explicit information-gain term. A robot can therefore keep moving toward regions with familiar anchors while leaving geometrically informative corridors underexplored. The authors identify this exploitation bias as future work, especially for environments where the useful next observation is not the region with the strongest current semantic signal.

Validating the Co-occurrence Prior

Navigation performance alone cannot isolate whether the room model is accurate, because perception, mapping, and control also affect success. The authors therefore measure co-occurrence directly from HM3D room annotations. For target $t$ and detected object $o$, ground-truth co-occurrence is:

$$P(t\mid o)=\frac{|\mathcal{R}_t\cap\mathcal{R}_o|}{|\mathcal{R}_o|},$$

where $\mathcal{R}_t$ and $\mathcal{R}_o$ are the room sets containing the corresponding instances.

Raw labels are mapped through Habitat's Matterport semantic table to 40 standardized categories. The evaluation removes abundant architectural categories such as floor, wall, and ceiling because they carry little discriminative room information. This leaves 21 object categories. The authors compare the ordering induced by room-probability metrics with the annotated conditional probabilities using per-target Spearman correlation, Recall@5, and NDCG@5.

This validation is stronger than showing a few visually plausible heatmaps. It tests whether the text-derived prior aligns with actual same-room frequencies in the evaluation scenes. It remains a dataset-level proxy rather than a robot experiment, but it separates the semantic representation from the navigation policy and exposes where the room model succeeds or fails.

Experimental Setup

The navigation experiments use Habitat and the HM3D semantic validation split: 2,000 episodes across 20 indoor scenes. The targets are chair, bed, potted plant, toilet, TV, and couch. Hardware consists of an Intel Core i9-12900K and an NVIDIA Quadro RTX 8000 with 48 GB of memory. The scenes and episodes are unseen, although the target categories are fixed for validation.

RPV-V1 and RPV-V2 share the room-probability model and perception schedule. V1 stamps signals with an isotropic Gaussian, while V2 uses free-space FMM propagation. This paired design isolates the contribution of topology from the contribution of the object relevance score. The baselines include VLFM, UIAP-OGN, zero-shot and trained L3MVN variants, and PIRLNav. PIRLNav is trained by behavior cloning on 77,000 human demonstrations from the HM3D training split, so it represents a different supervision regime rather than a direct zero-shot peer.

Co-occurrence Ranking Results

The full-row Spearman correlation for the RPV dot product is 0.329, slightly below the 0.355 to 0.365 range of several alternative distribution measures. The navigation objective, however, depends more on bringing the strongest anchors to the top of the list. The dot product achieves Top-5 Spearman 0.319, Recall@5 0.524, and NDCG@5 0.715, the best values among the compared metrics.

Validation measure	RPV dot-product result	Interpretation
Full-row Spearman	0.329	Overall rank agreement across all 21 classes is modest
Top-5 Spearman	0.319	The strongest candidate anchors retain usable ordering
Recall@5	0.524	More than half of strong co-occurrences enter the first five
NDCG@5	0.715	High-value anchors tend to receive high rank credit
Chair normalized entropy	0.773	A generic object has weak room specificity

Table 1: Co-occurrence validation reported in the paper's ranking analysis and Figure 5.

The lower full-row correlation is consistent with the multiplicative suppression built into the dot product. A room with little probability mass in either distribution contributes almost nothing, so weak associations collapse into a narrow low-score band. That behavior is beneficial when ranking the few anchors that can drive exploration, but it makes the complete ordering less faithful to small differences among unrelated objects. The square-root scale transform expands this band before propagation.

Generic objects expose the limit of the representation. A chair has normalized entropy 0.773 because chairs occur in many room types. Its distribution is broad, so the dot product admits many weak matches. Low-entropy objects such as a bed, bathtub, or toilet concentrate their mass and act as stronger functional markers. The authors do not treat entropy as an explicit penalty in the current score, which leaves room for false confidence around generic furniture.

Figure 5: Full-row and top-five rank agreement, Recall@5, and NDCG@5 for different room-distribution measures.

Figure 6: Room-probability vectors sorted by entropy. Specialized fixtures concentrate their mass, while generic objects spread across several room types.

Navigation Results

RPV-V2 reaches SR 0.539, SPL 0.307, and DTG 4.06 meters on 2,000 validation episodes. VLFM reaches SR 0.523 and SPL 0.303, so the relative improvements are 3 percent and 1.3 percent, corresponding to absolute changes of 0.016 and 0.004. The result is measurable but small. It supports the value of object-level, room-mediated evidence without showing that image-holistic methods have become obsolete.

UIAP-OGN records SR 0.523 and SPL 0.290. RPV-V2 is higher on both metrics and uses only raw object-label strings, with no ensemble of handcrafted prompts and no inference-time language model. That design reduces prompt sensitivity and keeps the semantic signal attached to segmented instances rather than a potentially cluttered whole image.

PIRLNav has the highest SR at 0.641, but its SPL is 0.271. Its 77,000 demonstrations buy more successful completions, while its paths are relatively less efficient. The comparison should not be used to claim that the training-free method beats supervised navigation overall. It instead shows that the room-mediated method offers a strong efficiency tradeoff under zero-shot constraints.

Method	Task training	SR up	SPL up	DTG down
L3MVN feed-forward	Yes	0.542	0.255	3.93
PIRLNav	Yes	0.641	0.271	Not reported
L3MVN zero-shot	No	0.504	0.231	4.43
UIAP-OGN	No	0.523	0.290	Not reported
VLFM local evaluation	No	0.523	0.303	4.18
RPV-V1	No	0.534	0.304	4.12
RPV-V2	No	0.539	0.307	4.06

Table 2: HM3D ObjectNav performance from the paper's Table I. DTG is reported in meters.

Propagation Ablation and Runtime

The switch from RPV-V1's isotropic Gaussian to RPV-V2's FMM propagation raises SR by 0.005, raises SPL by 0.003, and reduces DTG by 0.06 meters. The changes point in the same direction but are small. V1 can leak weak signal through walls into disconnected regions. Because frontier evidence is normalized, many broad weak responses can dilute a stronger cue. V2 constrains propagation to reachable free space and therefore makes a semantic response more likely to correspond to a physically related region.

V1 is not uniformly worse. Its topology-independent stamp can affect unknown space immediately after a detection, while FMM stops at the boundary of unknown cells. This may help V1 avoid an excessive bias toward dense nearby objects. The authors suggest combining isotropic and flood-fill signals as future work, which is reasonable because the two methods fail in different geometric situations.

RPV-V2 averages 335.5 ms of inference time per step. The number covers the complete decision path while full perception is throttled to every third exploration step. It is feasible in the experimental setup, but the cascade of Mask2Former, YOLOE, SAM 3, CLIP, and mapping remains substantial for a battery-constrained mobile robot. Offloading perception and vision-language inference to an edge server is the authors' proposed deployment pattern.

Implementation Correspondence

The official code follows the paper's decomposition rather than hiding the method in a monolithic policy. The room lexicon is explicit in vlfm/vlm/room_types.py:1-11. The embedding path in vlfm/vlm/clip.py:63-109 computes cosine scores, applies the default 0.03-temperature softmax, and multiplies the detected-object matrix by the target distribution. These operations correspond directly to the first three equations.

The policy combines target and detection probability vectors in vlfm/policy/itm_policy.py:252-310. It keeps only positive scored instances before converting their mask centroids and depth into world positions. The value map then uses scikit-fmm around vlfm/mapping/value_map.py:300-410, applies the square-root weight and adaptive sigma around lines 556-566, aggregates labels and averages nonzero evidence around lines 584-620, and sorts frontier candidates around lines 734-775.

A subtle correspondence is the neighborhood median. The paper describes the score at a frontier, but the implementation in vlfm/utils/img_utils.py:213-264 operates on a square image crop with a circular mask, removes zero entries, and reduces the rest with the default median. This matters for reproducing the system: the semantic statistic is local to each candidate waypoint, not a median over an entire connected frontier component.

Limitations

The authors explicitly note that RPV does not account for entropy. Generic objects such as chairs and tables produce broad room distributions, and their signals can dilute more specific evidence. Entropy-aware pruning, clipping, or regularization could penalize low-specificity anchors and increase the dynamic range between beds, bathtubs, and ambiguous furniture. The current square-root transform rescales low scores but does not estimate uncertainty directly.

The authors also acknowledge that their analytical co-occurrence estimate is only modestly correlated with annotated spatial relations. Full-row Spearman is 0.329 and Recall@5 is 0.524, so many true associations are still missing from the top five. Better statistical models or scene priors might improve this alignment, but they risk introducing the training dependence that the method is designed to avoid.

A third stated limitation is the absence of an explicit information-gain term. The agent exploits semantic evidence and may continue toward familiar but depleted regions instead of selecting a frontier that would reveal more free space. This can matter in long corridors, lobbies, and multi-exit scenes where the best next observation is not the strongest current semantic hotspot.

Independent limitations are also important. The nine-room lexicon is culturally and architecturally specific. Residential room names may not transfer cleanly to hospitals, warehouses, offices, or mixed-use buildings. The authors state that the lexicon can be replaced, but they do not run cross-domain experiments showing that CLIP text geometry and the new room taxonomy preserve useful priors after replacement.

The evaluation is simulation-only and covers six target categories. It does not measure calibration error, lighting changes, dynamic obstacles, detector drift, or physical robot latency. The cascade is computationally heavy, and an average of 335.5 ms per step does not expose peak memory, energy use, or tail latency. Those quantities determine whether the same pipeline remains usable on a mobile platform.

The underlying map is two-dimensional and single-floor. The paper reports that the method cannot solve 14.6 percent of HM3D validation episodes requiring multi-floor navigation. This is not a small implementation detail: stairs, ramps, and vertical connectivity change both reachability and the meaning of geodesic distance. Extending to multi-floor scenes requires a layered or graph-based topology, not simply copying the same value map onto every floor.

The gain over VLFM is also modest in absolute terms. SR rises by 0.016 and SPL by 0.004, while the propagation ablation changes them by only 0.005 and 0.003. These numbers are useful evidence for a component analysis, but they should temper claims about a broad replacement for image-level semantic grounding. The result is an interpretable improvement inside one benchmark and one controller family.

Conclusion

The paper's central contribution is an interpretable bridge between language-level object relatedness and physical room structure. A room-probability vector turns object labels into distributions over functional space. Their dot product favors mutually supported room evidence. Geodesic propagation then keeps that evidence on the navigable manifold, and frontier medians convert the resulting map into a search decision.

The empirical result is a modest but coherent improvement: RPV-V2 reaches SR 0.539 and SPL 0.307, ahead of the locally evaluated VLFM baseline in both metrics, while remaining training-free. The ablation and co-occurrence analysis show that topology and object-mediated evidence both matter, but the small deltas also expose the limits of the current analytical prior.

For robot developers, the reusable idea is the separation of roles. A compact semantic model proposes where evidence is likely to matter, an occupancy map determines which paths can carry that evidence, and a frontier statistic turns both into an action. The next meaningful step is not another temperature sweep. It is handling high-entropy objects, calibrating room vocabularies across domains, adding information gain, and representing multi-floor connectivity explicitly.

Golden quote: Semantic similarity says that two words belong near each other in language; room-probability overlap says that their physical evidence can meet in the same place.