NavJev: Efficient Vision-Language Navigation via Action-Centric Visual Compression and Discriminative Action-Semantic Memory

Paper Overview

Title: NavJev: Efficient Vision-Language Navigation via Action-Centric Visual Compression and Discriminative Action-Semantic Memory

Authors: Kai Sheng, Liuyi Wang, Jinlong Li, Haojie Dai, Chengju Liu, and Qijun Chen, College of Electronic and Information Engineering, Tongji University.

Links: arXiv:2609.34969 ｜ Project page

Code: The public NavJev-Code repository implements the core loop: waypoint extraction, parallel captioning and tagging, common-tag filtering, history construction, and typed Jev requests.

One-Sentence Summary

NavJev converts each candidate waypoint into a compact action profile containing geometry, BLIP context, and RAM tags, removes scene semantics shared by all candidates, remembers action-specific evidence across steps, and lets Jev make a typed probabilistic selection over that finite action set instead of repeatedly invoking multimodal autoregressive reasoning.

Navigation Setting and Action Space

At step t, the agent observes a panoramic RGB-D image composed of K views. The observation set is:

$$\mathcal{I}_{t}=\left\{(I^{\mathrm{rgb}}_{t,i},I^{\mathrm{depth}}_{t,i})\right\}_{i=1}^{K}.$$

A pretrained waypoint predictor converts that observation into a navigability heatmap. Non-maximum suppression extracts up to five candidates. Each waypoint has a relative heading theta, a relative distance d, and an associated image crop:

$$w_{t}^{i}=\left(\theta_{t}^{i},d_{t}^{i},I_{t}^{i}\right).$$

The action space also includes stopping:

$$\mathcal{A}_{t}=\{w_{t}^{1},\ldots,w_{t}^{M_{t}},\mathrm{Stop}\}.$$

This definition has consequences that show up directly in the efficiency experiments. NavJev performs waypoint-level macro-actions. Methods that predict low-level discrete moves execute 68.9 to 118.7 steps per episode, whereas NavJev averages 9.3 steps. A larger per-step call can still produce a much shorter episode when each call commits to a longer traversable segment.

Method: From Pixels to Action Profiles

Figure 2: NavJev pipeline. Panoramic RGB-D observations generate waypoint candidates; ACVC compresses each candidate into an action-centered profile; DASM removes common tags and organizes history; Jev emits a typed action.

flowchart LR
  A[Panoramic RGB-D] --> B[RGB and depth encoders]
  B --> C[Waypoint heatmap]
  C --> D[NMS up to 5 waypoints]
  D --> E[Action geometry
heading and distance]
  E --> F[ACVC batch]
  F --> G[BLIP captions]
  F --> H[RAM tags]
  G --> I[Action profiles Z]
  H --> I
  I --> J[Common tag intersection G]
  J --> K[Remove shared tags from each action]
  K --> L[DASM action-specific evidence]
  M[Executed action history] --> L
  L --> N[Structured state S]
  O[Navigation instruction] --> N
  N --> P[Jev typed action selection]
  P --> Q{Selected action}
  Q -- waypoint --> R[Move to theta and d]
  Q -- stop --> S[Terminate episode]
  R --> M
  

1. Keep executable geometry first

NavJev reuses the pretrained waypoint predictor from SmartWay. RGB features capture appearance; depth features capture traversable local geometry. The two modalities are fused by cross-attention and passed through a Transformer waypoint predictor:

$$\mathbf{P}_{t}=f_{\mathrm{wp}}\!\left(f_{\mathrm{rgb}}(\mathcal{I}_{t}^{\mathrm{rgb}}),f_{\mathrm{depth}}(\mathcal{I}_{t}^{\mathrm{depth}})\right).$$

In the released code, the heatmap is discretized into NUM_ANGLES=120 and NUM_CLASSES=12, so each sector is about three degrees. NMS then uses max_predictions=5. Those constants matter because they turn a continuous heatmap into a short, enumerable action set before language reasoning begins. They also make the eventual Jev schema clean: every legal option has a heading, a distance, and observable evidence.

2. Compress each candidate, not the whole panorama

Sending raw candidate crops to an MLLM at every step would still repeat visual encoding and generation. Instead, ACVC runs BLIP and RAM on the same image batch. BLIP produces a concise caption of the local context, while RAM produces semantic tags for salient objects and scene categories:

$$C_{t}^{i}=f_{\mathrm{BLIP}}(I_{t}^{i}),\qquad R_{t}^{i}=f_{\mathrm{RAM}}(I_{t}^{i}).$$

The two signals are complementary. A caption can say that the agent is looking into a large room with furniture; tags can name armchairs, tables, ceiling fixtures, hallways, elevators, or vending machines. The caption supplies continuity and spatial context. Tags supply discrete lexical anchors that can be matched against words in the instruction. NavJev combines both with the action's motion description and geometry:

$$Z_{t}^{i}=\left(m_{t}^{i},\theta_{t}^{i},d_{t}^{i},C_{t}^{i},R_{t}^{i}\right).$$

The motion field m is generated from the relative heading. In one_stage_prompt_manager.py::get_action_concept_backtrack(), the implementation maps relative angles into phrases such as turn sharp left, turn left, turn slight left, go forward, and their right-hand equivalents, using a 30-degree threshold. Therefore the action profile is not a loose image caption; it is a sentence-like description of an executable move. The full set is:

$$\mathcal{Z}_{t}=\left\{Z_{t}^{i}\right\}_{i=1}^{M_{t}}.$$

3. Make compression latency predictable

Because neighboring waypoints may project onto the same panoramic crop, ACVC deduplicates identical images and reuses the resulting BLIP and RAM outputs across actions. BLIP and RAM also run concurrently on the same batch. Thus the effective perception time is approximately the maximum of the two branches, not their sum:

$$T_{\mathrm{ACVC}}\approx\max\!\left(T_{\mathrm{BLIP}},T_{\mathrm{RAM}}\right)+\epsilon.$$

The code exposes this design concretely. In vlnce_baselines/common/base_il_trainer.py, the navigation loop creates a ThreadPoolExecutor and organizes CUDA streams for the two perception branches. The trainer later calls build_jev_request() in vlnce_baselines/navjev/jev_caption.py to serialize the structured options. In the measured simulation setup, the BLIP-RAM stage takes only 72.4 ms, 11.2% of the 647.9 ms average step. Perception is no longer the dominant cost.

4. DASM: remove common evidence before reasoning

Nearby candidates originate from the same scene. They often share ceilings, walls, furniture, corridors, floors, and broad room categories. Repeating those tags in every option increases prompt length without helping the decision. DASM therefore computes the common tag set:

$$G_{t}=\bigcap_{i=1}^{M_{t}}R_{t}^{i}.$$

G is not discarded. It is stored once as shared scene context. Each candidate then retains only the tags not shared by all actions:

$$\widetilde{R}_{t}^{i}=R_{t}^{i}\setminus G_{t},\qquad i=1,\ldots,M_{t}.$$

The discriminative action state becomes:

$$D_{t}^{i}=\left(m_{t}^{i},\theta_{t}^{i},d_{t}^{i},C_{t}^{i},\widetilde{R}_{t}^{i}\right).$$

The source implementation follows this arithmetic almost literally. _deduplicate_shared_ram_tags() normalizes tags with case folding, computes the intersection, and removes those tags from each option. build_jev_request() then places the filtered options, motion fields, geometry, captions, and shared context into a typed payload. This separation prevents the shared notion of "a room with furniture" from overwhelming the one candidate that actually contains an elevator or vending machine.

5. History as action-semantic memory

A trajectory log such as "forward, left, forward" tells the robot how it moved, but it loses the visual reason that made each move useful. NavJev augments history with the semantic profile of every executed action. Let phi(a_t) denote the executed action's motion description, heading, distance, BLIP caption, and filtered RAM tags. The history update is:

$$\mathcal{H}_{t+1}=\mathcal{H}_{t}\oplus\phi(a_{t}).$$

The current structured state consists of history, shared context, and the filtered current candidates:

$$\mathcal{S}_{t}=\left(\mathcal{H}_{t},G_{t},\left\{D_{t}^{i}\right\}_{i=1}^{M_{t}}\right).$$

This matters in scenes with repeated structures. If the robot has already passed a lounge with black sofas and long tables, that evidence can inform later comparisons with a similar-looking lounge. DASM therefore has two mechanisms: intra-step filtering reduces redundancy among current candidates, and inter-step retention preserves why past candidates were chosen. The ablation is strong evidence that the second mechanism is not decorative: adding DASM increases SR from 21.0% to 27.0% and SPL from 16.1% to 22.4%.

6. A typed decision instead of generated reasoning

Given the instruction and the structured state, Jev selects an option:

$$a_{t}=f_{\mathrm{Jev}}\!\left(\mathcal{L},\mathcal{S}_{t},\mathcal{A}_{t}\right),\qquad a_{t}\in\mathcal{A}_{t}.$$

If the selected option is waypoint j, the controller receives:

$$u_{t}=\left(\theta_{t}^{j},d_{t}^{j}\right),\qquad a_{t}=w_{t}^{j};$$

if it selects stop, the episode terminates. The schema has three consequences. First, the model cannot invent a direction that is not in the candidate set. Second, the output is probabilistic, so downstream systems can log uncertainty or add safety thresholds. Third, the decision latency is not inflated by decoding a variable number of reasoning tokens. The language instruction still requires semantic interpretation, but it is matched against compact action-bound evidence rather than raw pixels and free-form explanations.

7. Reading the released loop

The repository confirms that ACVC and DASM are not only conceptual layers. In jev_caption.py, each candidate crop is identified by the SHA-256 hash of its bytes before captioning. When two waypoint choices resolve to the same crop, the function builds the BLIP and RAM requests once and copies the result into both action profiles. This is the implementation of the paper's deduplicated ACVC batch. The same module defines build_jev_request() as a typed request: each option carries an identifier, motion phrase, heading, distance, caption, and filtered tag list, while the shared tags are placed in their own field rather than being repeated.

The intersection filter is deliberately small. _deduplicate_shared_ram_tags() case-folds the tag vocabulary, computes the intersection across every current option, and subtracts that set from each option. This means that a tag survives the candidate field only when at least one possible move differs from the others in that respect. It also means that changing the candidate set can change the discriminative representation: four doorway-like directions may jointly share a word, whereas two of them may retain it once other options disappear. That behavior is appropriate for a decision over the available set, although it makes the evidence less stable as a standalone scene description.

The prompt manager supplies the temporal half of DASM. In one_stage_prompt_manager.py, history is not merely a list of low-level commands; it appends the executed action concept, relative heading, distance, caption, and surviving semantic tags. The Trainer's main loop then follows the paper's order: waypoint prediction, crop hashing, parallel captioning and tagging, common-tag removal, construction of the structured state, and a Jev choice. The constants in the policy package also make the candidate space explicit: 120 angle bins, 12 distance or depth classes, and at most five NMS predictions. Thus the probability head is not choosing from an open-ended token sequence; it is choosing from a small set prepared by the geometry layer.

8. The deployment ledger

The efficiency numbers become practical when translated into machine placement. At 9.3 steps and 6.03 seconds per episode, one episode spends 534.6 ms on decision calls and about 72.4 ms on the BLIP-RAM batch. The peak perception-plus-policy footprint is 4.46 GiB. That is not the smallest possible robot stack, but it fits comfortably on an 8 GiB accelerator, whereas the compared VLN systems occupy roughly 17.5 to 35.8 GiB. The memory result can determine whether several agents share one local machine or whether each robot needs its own expensive GPU.

The $0.06 total cost is equally important, but its interpretation should stay structural rather than speculative. This paper's 100-episode bill is small because Jev is inexpensive and because macro-actions reduce the number of calls. That combination changes the cost curve for ablations, retries, fleet experiments, and training-data generation. It does not remove API dependencies. Since Jev accounts for most of the step latency, a team deploying NavJev must still handle service availability, regional latency, retries, and version drift.

The typed probability output is another deployment asset. A waypoint decision with low confidence can trigger additional observation, a conservative fallback, or logging for human review. The paper does not yet turn this uncertainty into a full safety policy, and none of its reported metrics measure rejection quality. Still, receiving a calibrated distribution over legal actions is a much cleaner starting point than parsing a free-form explanation and guessing whether the robot was confident.

9. How to read the accuracy tradeoff

The strongest NavJev comparisons are controlled. When GPT-4o and Qwen3.8-Max receive the same ACVC-DASM representation, Jev wins on SR, SPL, nDTW, latency, and cost. That comparison says the compact state contains enough action-relevant evidence to make a direct choice, and that explicit typed selection can outperform a large autoregressive decoder on this finite task. It does not say the representation is inherently superior to every prompt design, because those baselines had to consume the same state rather than their own richer visual input.

The broader system table requires a different reading. NavJev's 27.0% SR is below SmartWay's 29.0% and P2DNav's 50.0%. But NavJev's value proposition is not the highest accuracy at any cost. It keeps SPL close to SmartWay while changing the inference contract from multimodal generation to action selection. The efficiency table shows the tradeoff explicitly: a slower per-step call can still yield a much shorter episode if it commits to traversable macro-actions.

Supervised VLN-CE scores, often 60.5% to 65.0% SR, should not be merged into the same ranking. Those systems train on task data and use different evidence interfaces. NavJev is evaluated zero-shot on unseen environments. The meaningful question is whether the ACVC-DASM-Jev architecture preserves navigation competence at a deployable cost, and the answer is provisionally yes for short-horizon R2R-CE episodes.

10. Matching an instruction to action evidence

The Jev request keeps the instruction as a global route, while each option becomes a short criterion. The task description tells the model to use the instruction, executed history, previous planning, waypoint geometry, shared scene context, filtered RAM tags, and BLIP captions, while explicitly stating that images are not sent. The question then asks for exactly one action that best continues the route. This makes the semantic match auditable: an option is judged by its direction, metric distance, visible context, and position in the execution history, not by an unconstrained image description.

The released prompt material also shows why history must be semantic. A common navigation mistake is to see a referenced object again and assume the corresponding instruction clause is still unfinished. The prompt-level guidance warns the decision maker to verify completion from past movement, not merely from object visibility. DASM supplies that distinction by retaining what the robot observed when it moved. The state therefore has enough information to avoid looping toward an already visited landmark.

The code performs one final normalization from the text-option interface to the typed interface. It strips synthetic phrases such as navigation node and visual observation, separates shared RAM tags into a scene field, converts headings to degrees and distances to meters, and labels stop and backtrack actions specially. This normalization is easy to overlook, but it is central to reproducibility. The same instruction is paired with the same finite option set and the same evidence order at each step, so experiments can compare decision quality without trying to align several generations of model prose.

Experiments: Efficiency Without Destroying Competence

All simulation experiments use Habitat on the R2R-CE val-unseen split. Metrics include NE, OSR, SR, SPL, and nDTW. Simulation inference runs on an NVIDIA RTX PRO 6000 GPU; Jev is accessed through its official API. Real-world tests use a wheeled robot with an Insta360 X4 panoramic camera. Candidate extraction follows CLASH's clustering strategy, while ACVC, DASM, and Jev remain unchanged from simulation.

Comparison on R2R-CE

Compared with Open-Nav GPT-4o, NavJev improves SR by 8.0 points and SPL by 6.3 points. Compared with Open-Nav Llama3.1-70B, the improvements grow to 11.0 and 9.5 points. NavJev nearly matches SmartWay's SPL while avoiding multimodal reasoning. It remains below P2DNav, which reaches 50.0% SR and 30.6% SPL, so the fair reading is that NavJev makes zero-shot navigation much cheaper and faster while staying competitive, not that it surpasses the strongest zero-shot navigation systems.

End-to-end efficiency

This table must be read along three axes. Per step, NavJev is not the fastest; NaVid, NaVILA, and StreamVLN are faster. Per episode, NavJev wins because 9.3 macro-actions replace dozens of low-level calls. In memory, NavJev requires 4.46 GiB, far below the compared models. The paper also reports speedups of about 41.5 times, 8.2 times, and 7.6 times over Open-Nav, SmartWay, and P2DNav respectively.

The latency breakdown is equally informative. Of 647.9 ms per step, Jev accounts for 534.6 ms, BLIP-RAM for 72.4 ms, waypoint prediction for 25.1 ms, and environment interaction for 14.9 ms. Once the visual interface has been compressed, the next engineering target is no longer the perception model but the remote typed-decision call. Localizing Jev or reducing network round trips would likely matter more than swapping another image encoder.

Same representation, different decision models

This controlled comparison isolates the decision operator. All models receive the same ACVC-DASM state and choose from the same candidates; autoregressive reasoning is disabled for GPT-4o and Qwen3.8-Max. Jev raises SR above Qwen3.8-Max by 2.0 points and above GPT-4o by 13.0 points. Its SPL exceeds Qwen3.8-Max by 6.4 points and nDTW reaches 43.5, the best in the group. Meanwhile, the cost is only about 2.7% of Qwen3.8-Max and roughly 1.6% of GPT-4o. OSR falls to 35.0, so the gain is not stronger coarse goal detection; it is better constrained selection along the instructed trajectory.

Component ablation

RAM raises SR from 19.0% to 21.0% and SPL from 14.4% to 16.1%, but trajectory length worsens from 12.10 to 15.83. Adding DASM raises SR by 6.0 points and SPL by 6.3 points while returning trajectory length to 11.58. More tags alone do not improve the decision interface; separating shared scene context from action-specific evidence does.

Real-world deployment

Figure 3: Two real-world tasks expose the actual per-step state: motion, heading, distance, BLIP caption, and RAM tags. The first trajectory stops after vending machine appears; the second stops after elevator evidence is visible.

Each environment contains ten tasks. NavJev reaches average OSR of 60.0% and SR of 50.0%, while Qwen3.8-Max reaches 35.0% on both. Decision latency falls from 1.10 s to 0.65 s and step time from 1.23 s to 0.79 s. The sample size is small, but the experiment checks that the compact representation works with real panoramic video, clustered candidates, and physical motion—not only with Habitat's controlled simulator.

Where NavJev Fails

Figure 4: Among 100 episodes, 27 succeed and 73 fail. 60 failures come from incorrect waypoint selection, 7 enter the wrong region, 4 stop early, and 2 diverge. The right cases show chosen directions that look semantically similar to the oracle direction.

The failure distribution is unusually clear. Of 73 failures, 60, or 82.2%, are caused by selecting the wrong waypoint. Seven enter the wrong region, four stop too early, and two diverge persistently. The system's weakness is therefore not stop detection or long-range planning collapse; it is the local discrimination between visually similar candidate directions.

The representative examples make the limitation concrete. Instructions mention relations such as "to the left of the stairs" or "through the second hallway." The chosen and oracle viewpoints share many cues, and the filtered tag sets still leave similar words. The agent may select a direction that is semantically plausible but geometrically wrong.

This is the expected cost of a text-based action profile. ACVC preserves action-related semantics and coarse geometry, but it loses dense pixel relationships: exact relative angles, occlusion order, fine texture differences, and subtle metric shifts. When the instruction requires those relations, captions and tags become too coarse for reliable discrimination.

Limitations and Engineering Implications

First, the authors identify fine-grained spatial reasoning as the main bottleneck. The 60/73 concentration in wrong waypoint selection supports that diagnosis. A better ACVC should not simply shorten text further; it should encode relational geometry among landmarks, viewpoints, and instruction references.

Second, Jev is an external API call and dominates 82.5% of measured latency. Typed selection removes token-generation overhead but adds service availability, network variability, queueing, and billing concerns. For long-duration deployment, local inference or redundant decision services become first-order engineering requirements.

Third, the visual interface is lossy. BLIP and RAM provide good semantic recall for rooms, furniture, doors, elevators, and vending machines, but they do not reconstruct precise object poses. If two directions differ only by whether the reference object is on the left or right edge of the view, the current state may not carry enough evidence.

Fourth, the evaluation breadth is limited. The controlled decision-model comparison, ablation, and failure analysis use a 100-episode R2R-CE subset; real-world evaluation has 20 tasks. NavJev's 27.0% SR remains below task-fine-tuned systems at 37.4% to 65.0% and below P2DNav's 50.0%. The contribution is a large reduction in closed-loop cost while preserving usable zero-shot performance, not a universal accuracy record.

Conclusion

NavJev makes a precise architectural claim: embodied navigation does not need to re-interpret raw images at every step if the state is organized around executable actions. ACVC supplies that state by binding waypoint geometry to BLIP context and RAM tags. DASM makes the state discriminative by removing common semantics and preserving action-specific evidence across time. Jev closes the loop with a typed probabilistic choice over the known action set.

The most useful lesson for robot system builders is that interface design changes both cost and reliability. NavJev does not merely replace one large model with smaller perception models. It turns a high-dimensional generation loop into a finite-state selection loop, and its measurements show how that choice reduces episode time, GPU memory, and API expense. The next version of this paradigm will likely depend on stronger relational spatial encoding and a localizable decision backend.

Quote

"A closed-loop navigator does not need to look at the entire image again; it needs to remember why this direction is different."