Interactive Navigation Beyond Reachability: Benchmark and Agentic Framework

Paper: Interactive Navigation Beyond Reachability: Benchmark and Agentic Framework (arXiv:2610.17089v1, cs.RO, 2026-10-12)
Code: github.com/ZGCA-HMI-Lab/MultINav · Project page: zgca-hmi-lab.github.io/MultINav · Dataset: huggingface.co/datasets/Piqiuni/MultINav-Bench

One-Sentence Summary

MultINav puts interaction back into navigation: a closed door can sever the only route to a goal, and a closed fridge can hide a target that is meters away — the paper unifies channel interactions (connectivity), container interactions (visibility), and their sequential dependencies, releases the 3,000-episode MultINav-Bench, and proposes a three-stage agent over a persistent interaction graph that lifts mixed-task success from MoMa-LLM's 1.7% to 24.1%.

Background and Motivation

PointGoal, ObjectGoal, and vision-and-language navigation have driven sustained progress in embodied navigation. These tasks test localization, mapping, search, and instruction following, but they generally assume that traversability and object states remain fixed within an episode. The abstraction omits a basic property of indoor environments: interaction changes both the space a robot can traverse and the targets it can observe.

A closed door could disconnect the target room or force a long detour; a closed refrigerator, cabinet, or drawer could conceal a nearby target. These effects can also be sequential: opening a door permits entry into a room, where opening a container then reveals the target. An agent must therefore reason about how each interaction changes the environment and enables subsequent navigation, exploration, or interaction, rather than treating interaction as a side effect of path execution.

Prior work laid foundations: Interactive Gibson studies movable obstacles and the trade-off between path efficiency and physical disturbance; Interactive Visual Navigation measures free-space recovery; CaMP brings interaction into PointGoal navigation. The shared gap is task-level judgment — does an interaction restore connectivity, reveal a target, or is it simply unnecessary? Existing benchmarks evaluate either reachability or disturbance, never interaction necessity itself, and never place connectivity changes and visibility changes inside one task definition.

The paper formalizes this gap as the MultINav problem: unify channel interactions that change spatial connectivity, container interactions that change target visibility, and their sequential dependencies, demanding a navigation-decision framework that reasons about both interaction types and their ordering jointly. The agent must answer four questions at once: whether to interact, which object to select, when to act, and how to exploit the resulting state change to proceed toward the goal.

This positioning distinguishes MultINav both from non-interactive navigation and from interaction for interaction's sake: opening a door does not constitute success — open the door, fail to locate the target, and the episode still fails; conversely, most of a scene's affordances are not worth performing, and superfluous attempts are charged as cost. An interaction's value is measured only by its contribution to reaching or observing the target.

MultINav-Bench Design

MultINav-Bench is a diagnostic ObjectGoal benchmark: 3,000 frozen lightweight episodes in MolmoSpaces spanning 639 houses, balanced across channel / container / mixed splits (1,000 each), with explicit interaction-necessity labels, distractor objects, and oracle demonstration paths. It evaluates whether an agent reaches the goal while purposefully deciding when and what to interact with.

Task and interaction taxonomy. The embodied state is $s_{t}=(x_{t},M_{t},z_{t})$: robot state $x_{t}$, geometric/semantic map $M_{t}$, and articulation states $z_{t}$ of interactive entities. Actions split into navigation $\mathcal{A}_{\mathrm{nav}}$ (updating $x_{t}$) and physical interaction $\mathcal{A}_{\mathrm{int}}$ (updating $z_{t}$, thereby modifying traversability or target visibility). Each episode specifies the scene $\mathcal{S}_{i}$, initial states $(x_{i}^{0},z_{i}^{0})$, goal object $g_{i}$, annotated necessary interactions $\mathcal{I}_{i}^{\star}$, an oracle plan $\pi_{i}^{\star}$, and a ground-truth necessity label $y_{i}\in\{\texttt{required},\texttt{unnecessary}\}$.

Terminal success binds distance and visibility: $S_{i}=\mathbb{1}[d_{i}^{\mathrm{pass}}\land v_{i}^{\mathrm{pass}}]$ — stop within the planar distance threshold with the target visible in the head camera. Physical interaction enables state changes but does not establish terminal success: opening a door without locating the target yields $S_{i}=0$. Intermediate process is measured separately by ISR, IP, and normalized total cost.

Three task types. Channel interactions (hinged doors) govern spatial reachability, featuring critical closed doors (required), distractor doors (unnecessary), and composite layouts. Container interactions (refrigerators, cabinets, drawers) expose occluded targets, enabling the visibility required for navigation completion. Mixed episodes mandate sequential composition — traversing a closed door to reach a closed refrigerator. Across the 3,000 episodes, 2,654 are labeled required and 346 are unnecessary negative controls, allowing unnecessary interaction attempts to be measured without treating all exploration as an execution error.

Data generation. Starting from ProcTHOR-10K houses and NavToObj seeds, candidates are synthesized via counterfactual checks: closing a channel must sever the shortest route to the goal; concealing a target inside a container must invalidate initial visibility until opened. Candidates pass articulation readback, target-identity verification, and oracle rollout validation before freezing 1,000 balanced episodes per domain. Beyond the frozen light evaluation split, the pipeline supports a full collection mode recording synchronized egocentric/allocentric visual streams, continuous base trajectories, 6D end-effector targets, and articulation states in unified HDF5 records — an expert pretraining corpus for vision-language-action (VLA) models.

Figure 2: Construction and composition of MultINav-Bench. Top: channel interactions alter reachability (with a route-shortening subset), container interactions alter visibility and accessibility. Bottom: construction and collection pipeline. Right: required-interaction counts and feasible-route lengths across the 3,000 episodes.

Five metrics. The protocol decouples terminal goal achievement from intermediate interaction efficiency: SR (terminal ObjectGoal success), SPL (success-weighted planar path efficiency relative to a feasible oracle route), ISR (mean fraction of verified necessary interaction effects on required episodes, partial credit for partial completion), IP (category-level interaction precision, the episode-mean fraction of attempts producing a verified physical effect in a target interaction class), and Total Cost $\min((L_{i}+0.3A_{i}+E_{i})/30,1)$ (path length $L$, attempts $A$, non-qualifying attempts $E$; cost is 1 on failure; the normalization constant 30 is fixed before evaluation).

The MultINav-Agent Framework

The agent maintains a persistent, state-conditioned interaction graph $\mathcal{G}_{t}=(\mathcal{V}_{t},\mathcal{E}_{t})$ that connects semantic perception, action selection, and post-interaction verification in a loop: verified changes update the graph, and the updated graph guides the next navigation, exploration, or interaction.

Figure 3: Overview of the MultINav-Agent framework.

Stage 1: semantic interactive perception. Graph nodes are rooms, interactive entities, objects, and the robot; edges are connectivity, containment, and visibility relations. Perception associates entities across observations, and visual attribute inference estimates interaction class and coarse state (open, closed, uncertain). Interactive nodes also retain candidate operations, observation confidence, and recency. The graph is state-conditioned because relation validity depends on the observed interaction state: a closed door blocks its mediated room connection, whereas a verified opening permits traversal; opening a container permits inspection of its interior, but target visibility is established only when the target is observed. Newly observed entities extend the graph, while previously observed entities persist beyond the current view — stored states and relations provide context for later decisions, with confidence and recency distinguishing historical estimates from fresh evidence.

Stage 2: effect-guided action selection. At each decision point, the graph and navigation map generate candidates for navigation, frontier exploration, and interaction. An interaction candidate specifies an entity, operation, and approach viewpoint; eligibility checks use route feasibility, available visual evidence, and state reliability — these constrain currently executable candidates, while remembered entities remain in the graph even when no interaction with them is currently eligible. Each interaction candidate carries a coarse expected effect derived from its interaction class and graph relations: opening a channel may enable traversal, while opening a container may reveal its contents. This describes a possible change without assuming an unseen target is present. A vision-language model (Qwen3.6-35B-A3B-FP8) receives the goal, relevant subgraph, candidate states and effects, route costs, and decision history, and ranks eligible actions by expected contribution to reaching or observing the target, accounting for route prerequisites, plausible target locations, and previous attempts. Navigation and exploration remain available when interaction offers little task progress.

Stage 3: interaction verification and graph updates. For a selected interaction, the agent reaches the approach viewpoint, grounds the operable region and operation from the current view, and navigation/interaction skills execute the specification. Post-interaction verification reconciles fresh visual observations with execution feedback to assess the achieved state and expected graph effect: confident evidence updates the affected entity state, while conflicting or occluded evidence is marked uncertain. Channel updates revise room connectivity; container updates incorporate newly observed objects and their containment and visibility relations through perception. Failed or uncertain attempts remain in decision history to discourage immediate repetition. The updated graph regenerates candidates for the next round.

This loop supports sequential interaction dependencies: opening a door enables entry into a room; observations there can introduce a container candidate; verified container opening exposes objects whose observations support the next navigation decision. Each step composes connectivity and visibility changes during exploration:

graph TD
  START["Decision point t"] --> PERC["Semantic interactive perception
entity association / state & confidence"]
  PERC --> GRAPH["Persistent interaction graph G_t=(V_t,E_t)
rooms·doors·containers·objects / conn·contain·vis"]
  GRAPH --> CAND["Candidate generation
nav / frontier / interaction(entity+op+viewpoint)"]
  CAND --> FILTER["Eligibility checks
route feasible · visual evidence · state reliable"]
  FILTER --> VLM["VLM effect-guided ranking
expected connectivity/visibility contribution"]
  VLM --> SEL{"Interaction selected?"}
  SEL -- "No (low progress)" --> NAV["Navigate or frontier explore"]
  SEL -- "Yes" --> EXEC["Approach viewpoint → ground → execute"]
  EXEC --> VERIFY["Post-interaction verification
visual obs × execution feedback"]
  VERIFY --> UPDATE{"Evidence confidence"}
  UPDATE -- "High" --> APPLY["Update entity state
channel→connectivity / container→contents merged"]
  UPDATE -- "Conflict/occlusion" --> UNC["Mark uncertain"]
  UNC --> HIST["Record in decision history
failed attempts not repeated immediately"]
  APPLY --> HIST
  HIST --> GRAPH
  NAV --> GRAPH

A case study walks the loop end to end: the robot opens two doors to expand reachability, navigates to the refrigerator, and opens it to expose the target; per-step insets show how verified state updates guide continuous replanning. Figure 4's right panel shows the final interaction graph and trajectory (blue: navigation/exploration; orange: interactions).

Figure 4: Sequential interactions and graph updates. Opening two doors to reach the refrigerator, then opening it to reveal the target; insets show the evolving semantic maps and interaction graphs.

Experiments

Setup. Non-interactive evaluation runs on MolmoSpaces NavToObj; interactive comparison runs on MultINav-Bench. The benchmark provides raw image-based perception and a restricted ground-truth (RGT) perception mode; the primary comparison uses the same RGT interface for both interactive methods (episode-local object IDs, categories, and spatial observations for currently visible objects), reducing perception and instance-association errors as a confound while keeping exploration, interaction selection, state updating, and terminal verification as agent decisions. Hidden objects, simulator joint metadata, and oracle plans are withheld. The interactive baseline is a MolmoSpaces adaptation of MoMa-LLM; both share the same interaction executor and the Qwen3.6-35B-A3B-FP8 high-level decision model.

Non-interactive NavToObj. Without interaction, MultINav-Agent achieves 53.0% SR on MolmoSpaces NavToObj (2,000 tasks, ProcTHOR domain), above RING's 48.0% (leaderboard) and DualVLN's 21.0% (Habitat domain); locally evaluated SPOC scores 20.15% and VLFM 11.90%. The paper notes all baselines execute valid commands; the transfer gap stems from continuous control shifts lacking sliding contact feedback.

Main interactive results. On MultINav-Bench, MultINav-Agent reaches 55.80%, 45.00%, and 24.10% SR on Channel, Container, and Mixed tasks, exceeding MoMa-LLM by 36.20, 34.70, and 22.40 percentage points respectively:

Method / Split	SR (%) ↑	SPL ↑	ISR (%) ↑	IP (%) ↑	Cost ↓
MoMa-LLM · Channel	19.60	—	54.28	—	—
MultINav-Agent · Channel	55.80	higher	60.09	higher	lower
MoMa-LLM · Container	10.30	—	2.00	2.37	—
MultINav-Agent · Container	45.00	higher	46.70	38.91	lower
MoMa-LLM · Mixed	1.70	—	25.37	—	0.9951
MultINav-Agent · Mixed	24.10	higher	45.07	higher	0.9493

Table 1: Main results on MultINav-Bench by split. Container tasks show the largest ISR gap (46.70% vs 2.00%) and IP gap (38.91% vs 2.37%).

On Channel tasks the two methods' ISR differs by only 5.81 points (60.09% vs 54.28%) while SR differs by 36.2 — the metric design cleanly exposes the gap between "achieving an interaction effect" and "reaching the final goal". Container tasks show the largest ISR gap (46.70% vs 2.00%): MoMa-LLM barely learns to open the right container. Mixed tasks are hardest for both: MoMa-LLM reaches 25.37% ISR but only 1.70% SR; MultINav-Agent's 45.07% ISR (with partial credit) backs only 24.70% of episodes that complete all necessary effects.

Interaction-to-goal gap. ISR averages partial completion, so it need not track terminal SR. For MultINav-Agent: 393/654 required Channel episodes complete all necessary effects, of which 297 (75.6%) reach the goal; Container completes 466/1,000, of which 435 (93.3%) reach the goal; Mixed completes 247/1,000, of which 223 (90.3%) reach the goal. On Container and Mixed, the prominent difficulty is assembling the necessary interaction chain; Channel retains a larger navigation gap after effects are achieved.

Efficiency and failure patterns. MultINav-Agent has higher SPL and lower normalized Cost in every split, yet both methods incur near-ceiling Cost on Mixed tasks (0.9493 vs 0.9951) — failures cost 1, so the 759 Mixed failures contribute 0.759 to the mean. Exploration stalls are also telling: 516 on Mixed, 304 on Container, 158 on Channel, motivating further analysis of progress across interacting subgoals. The paper cautions that these terminal labels do not establish a causal failure mechanism.

Ablation. On a fixed 200-episode subset of the 1,000 Mixed tasks spanning 165 houses (sampled proportionally without replacement by container class and oracle-restored reference path length; the episode list frozen before evaluation):

Variant	SR (%) ↑	ISR (%) ↑	IP (%) ↑	SPL ↑	Cost ↓
Full model	23.0	41.8	40.4	0.107	0.961
Flat object memory	21.0	43.8	26.7	—	—
Greedy selection	22.0	—	36.9	0.134	0.933
Perception-only update	5.5	26.6	36.6	0.033	0.988

Table 2: Ablation study. Perception-only update causes the largest drop (SR 23.0% → 5.5%); the graph memory's clearest advantage is interaction precision, not required-effect completion.

The three variants expose different trade-offs rather than uniform gains. Flat object memory drops IP from 40.4% to 26.7% while raising ISR from 41.8% to 43.8% — the graph memory's clearest advantage is interaction precision: relational context helps select effective target-class interactions, while higher partial progress need not translate into goal completion. Greedy selection attains higher SPL and lower Cost with slightly lower SR and IP; the full model completes only two additional episodes out of 200, so semantic selection establishes no decisive success advantage — it trades a completion-efficiency trade-off. The strongest evidence comes from state updating: perception-only update collapses SR from 23.0% to 5.5% (17.5 points) with SPL down to 0.033 — without execution-informed graph updates, channel and container coordination falls apart. The paper cautions that the smaller differences for memory and selection should be interpreted cautiously on this fixed subset.

Real-world demonstration. In an indoor environment, a Unitree Go2 quadruped searches for an apple: starting near a closed door, the door is opened with human assistance, enabling access to the adjacent space; the robot explores the newly accessible area, extending its semantic map and interaction graph, then navigates to the refrigerator, which is also opened with human assistance so it can inspect the contents for the target apple. One object-search task links two distinct state changes: door opening changes connectivity, refrigerator opening exposes previously occluded contents.

Figure 5: Real-world navigation to find an apple with Go2. Door and refrigerator opening are human-assisted; map markers indicate navigation (blue) and interaction (orange).

Limitations

Author-stated limitations. First, the smaller SR differences for memory and selection should be interpreted cautiously on the fixed ablation subset rather than as statistically significant gains. Second, ISR's partial-completion credit means it need not track terminal SR — Mixed ISR of 45.07% includes partial credit while only 24.70% of episodes complete all necessary effects. Third, Cost is near ceiling on Mixed tasks (0.9493); aggregate improvements reflect both completion and trajectory efficiency rather than establishing shorter paths or cheaper execution conditional on success. Fourth, terminal labels such as exploration stalls do not establish a causal failure mechanism.

Perception mode of the main comparison. The primary comparison uses the restricted-GT mode to strip perception and instance-association errors. The benefit is a clean measurement of interaction representation and decisions; the cost is that conclusions do not directly transfer to pure vision input — the full image mode is left as an open problem. In real deployment, visual attribute inference and state estimation errors enter the graph update pipeline directly.

Human assistance in the real-world demo. Door and refrigerator opening in the Go2 demonstration are human-assisted, so the interaction execution layer is not closed-loop on hardware; additionally, benchmark scenes are synthesized from ProcTHOR-10K houses, which remain a distance from the clutter and physical door diversity of real homes.

Conclusion and Outlook

MultINav's contribution is double. On the benchmark side, interaction necessity becomes a first-class citizen: three task types (channel, container, mixed), required/unnecessary labels, five decoupled metrics (SR/SPL/ISR/IP/Cost), and an automatable data-generation pipeline give "should I interact during navigation" a systematic measurement for the first time. On the agent side, the three-stage framework hangs perception, selection, and verification on one persistent interaction graph; verified state changes drive the next candidate generation, so sequential dependencies compose loop by loop. Ablations show the graph memory's value in interaction precision (IP 40.4% vs 26.7%) and execution verification's value in task success (SR 23.0% vs 5.5%).

Next steps are clear: robustness under the full image perception mode, the subgoal progress problem behind the 24.1% mixed-task success rate (516 exploration-stall episodes), closed-loop interaction execution on hardware, and using the HDF5 expert trajectories from full collection mode for VLA pretraining. The next step in navigation likely lies in robots no longer treating doors and drawers as obstacles, but as writable world state.

Golden line: "Opening a door is not success; finding the target is. But never opening it may mean never finding it." MultINav measures an interaction's value by its contribution to the goal — most doors should stay closed; the few that must open decide the whole episode.