PAPER DEEP DIVE
GuidedNav: Shaping Vision-Language Navigation Representations through Pre-action Attention and Spatial Guidance
GuidedNav shapes vision-language navigation representations during training with subtask, landmark, and spatial guidance. It aligns pre-action attention with the active instruction segment and relevant visual regions, then distills long-horizon geometry and local spatial structure into a monocular RGB policy. The method improves success rates on R2R-CE and RxR-CE val-unseen while avoiding intermediate textual reasoning and online geometric encoding at inference.
Paper Record and Central Claim
Paper: arXiv:2610.13122v1, HTML full text.
Authors: Jingyu Guo, Jiaxin Huang, Chengxing Lin, Xiaoyu Luo, Cheng Wen, Shanshan Ye, Tingjin Chu, Shaoli Huang, and Mingming Gong.
Code: The authors provide the Guided-Nav repository and a project page. At commit 1d7e33fed6b489560491b4b6f3e912884fc14790, the repository contains a README and logo assets only. Checkpoints, inference code, datasets, and training code are explicitly marked as coming soon. The implementation discussion below therefore follows the paper and its appendix, not absent source code.
Central claim: GuidedNav improves monocular vision-and-language navigation by shaping the representation at the position that predicts the first action. It teaches that position to track the current instruction span, ground relevant landmarks, and absorb geometry from historical views, while preserving direct action generation at deployment time.
Why Another Navigation Objective Is Needed
A realistic route instruction is a sequence of local goals. The sentence may ask an agent to approach a yellow chair, pass a dining table, and enter a room. A large multimodal model can understand every word and still make the wrong decision because the words, the current object, and the spatial layout are not coupled at the right point in the computation. Navigation failures often emerge as progress errors: the agent continues to fixate on an old landmark, treats a future instruction as active, or recognizes a destination but approaches it from an inefficient direction.
Recent systems use intermediate reasoning, future-frame prediction, auxiliary questions, or explicit progress descriptions to add structure. These approaches can supply useful learning signals, but they also create a deployment question. Must the robot generate an explanation before every action? Does the action depend on a text trace that may be plausible but wrong? GuidedNav begins from a narrower and more operational premise: the context immediately before the first action token should already encode what to do, where to look, and how the local scene is organized. If it does not, can training objectives shape that context without adding another inference stage?
The paper separates the problem into linguistic progress and visual geometry. The first is represented by an active subtask span in the instruction. The second is represented by landmark masks in the current image and by history-conditioned spatial features. Both are training-time annotations or teacher signals. Neither is required from a user or a deployed robot.
This separation is useful because a single navigation loss gives weak credit assignment. A successful route confirms that the final action sequence was correct, but it does not say whether the model attended to the right phrase, grounded the right object, or used the right depth cue. Dense auxiliary objectives can provide that credit as long as they remain connected to the representation that controls action. The central design decision in GuidedNav is to read native attention at the pre-action position rather than create a separate interpretability branch that the policy can ignore.
A second design choice is to move expensive geometric supervision out of the inference graph. A frozen VGGT teacher can compare a current view with sampled history and expose rich spatial structure. Repeating that computation for every robot action would be costly. GuidedNav instead distills the teacher into current visual features and a local temporal fusion path. The robot carries the learned consequence of geometry, not the teacher itself.
Figure 1: The paper illustrates three forms of guidance on one route. The active instruction phase, its visible referent, and the surrounding spatial structure provide complementary signals.
Problem Formulation and Architecture
At decision step $t$, the policy receives an instruction $I$, sampled historical observations $H_t$, and the current image $o_t$. It predicts a short sequence of primitive actions:
$$A_t \sim \pi_\theta(\cdot\mid I,H_t,o_t),\qquad A_t=(a_{t,1},\ldots,a_{t,m_t}),\quad m_t\le 4.$$
The available primitives are forward movement, left turn, right turn, and stop. An action chunk terminates early when stop appears. Chunking lowers the number of calls to the language model, but it also means that each call must preserve the task state across several environmental updates. The model cannot rely on a new natural-language question at every primitive action.
GuidedNav starts from Qwen3-VL-8B-Instruct. Instruction tokens, historical visual tokens, current visual tokens, and action tokens enter an autoregressive decoder. During training, three guidance families are added around this backbone: subtask attention loss, landmark attention loss, and spatial guidance. The first two supervise attention read from the position immediately before the first action token. The third aligns hidden states with a frozen geometry teacher and supervises a temporal fusion residual.
flowchart LR I[Instruction] --> IT[Instruction tokens] H[Past RGB frames] --> VG[Current visual grid] O[Current RGB frame] --> VG VG --> TF[Sparse temporal fusion
three previous frames] IT --> PA[Pre-action query] TF --> PA PA --> SA[Subtask span loss] PA --> LA[Landmark mask loss] PA --> AC[Action tokens] H --> GT[Frozen geometry teacher] O --> GT GT --> LG[Long-horizon alignment] GT --> LR[Local residual target] LG --> PA LR --> TF AC --> ENV[Robot action chunk]
The important split is between training and deployment. The frozen teacher, auxiliary projectors, subtask annotations, and landmark masks disappear at inference. Sparse temporal fusion remains, and the language model still emits actions directly. This makes GuidedNav different from systems whose accuracy depends on generating an intermediate sentence or running a geometry model at every step.
Figure 2: Native pre-action attention connects instruction progress and visual grounding. Spatial guidance distills history-conditioned geometry into the policy, while temporal fusion remains available after the teacher is removed.
Reading the Decision Point
The model reads attention at the existing position immediately before the first action token. Let $Q_p$ denote its query. $K_I$ and $K_t$ denote the keys of instruction tokens and current-frame visual tokens. For a selected set of heads $\mathcal B$, the instruction or visual attention map is:
$$\alpha_D=\frac{1}{|\mathcal B|}\sum_{b\in\mathcal B}\operatorname{softmax}\!\left(\frac{Q_p^{(b)}(K_D^{(b)})^\top}{\sqrt{d_b}}\right),\quad D\in\{I,t\}.$$
Here $d_b$ is the per-head dimension. The implementation uses decoder block 27, counting layers from one, and heads 0 and 16 with zero-based indexing. Positional encodings remain active. Softmax is computed separately inside the instruction group and inside the visual group before averaging the selected heads.
Group-wise normalization matters. A joint softmax over a long instruction and hundreds of image tokens can make a localization score depend on the relative lengths of the two modalities. Separate normalization ensures that the language score describes competition among instruction words and that the image score describes competition among image patches. The maps are therefore measures of relative attention within each domain, not a common probability allocation across text and pixels.
Reading before the action token also avoids contaminating the supervised context with actions already generated. If attention were extracted after the model wrote part of the action chunk, the measured state would include its own output. GuidedNav instead targets the query that is about to decide the first primitive. No extra query token or special attention head is introduced, so the readout observes the backbone's native routing rather than a parallel diagnostic network.
Supervising Language Progress and Visual Reference
For the active instruction span $\mathcal S_t$, the subtask objective maximizes the fraction of instruction attention that lies on the current span:
$$\mathcal L_{\mathrm{Subtask}}=-c_t\log\left(\sum_{i\in\mathcal S_t}\alpha_{I,i}\right).$$
It does not prescribe a distribution across the span. A phrase such as "turn right at the kitchen" contains several useful words, and their relative importance can change with the view. The loss asks only that the action-producing position access the active phrase as a whole. The coefficient $c_t$ counteracts length imbalance. R2R intervals are grouped by trajectory and subtask, and the total weight of each group is balanced so that longer intervals do not dominate solely because they contain more examples.
Landmark guidance connects the active language span to visual evidence. Accepted masks are area-pooled onto the current visual-token grid. If $m_{t,j}\in[0,1]$ denotes the occupancy of patch $j$, the objective is:
$$\mathcal L_{\mathrm{Landmark}}=-\log\left(\sum_{j=1}^{N_t}m_{t,j}\alpha_{t,j}\right).$$
The mask is not normalized into a target distribution. This distinction is deliberate. Distribution matching forces attention to imitate the shape of an automatically generated mask, including errors at object boundaries. Area-supported attention asks the model to put mass on the accepted region while leaving the internal pattern flexible. In the appendix comparison, this attention-mass objective reaches $62.6$ success and $72.1$ oracle success, exceeding distribution matching and access regularization. The base model still has the best SPL, which shows that stronger grounding does not automatically eliminate every inefficient route.
Both losses operate on the same query. A sample can omit the subtask term if its span annotation is unreliable, and it can omit the landmark term if no mask passes filtering. Action-token cross entropy remains active. This asymmetric masking is important because the data engine uses language models, object detectors, and a segmentation model; one unreliable auxiliary label should not remove an otherwise valid navigation trajectory.
Distilling Spatial Structure
A monocular image does not fully determine scale, occlusion, or the relation between a visible surface and the path behind it. GuidedNav uses a frozen VGGT-1B teacher to produce history-conditioned features for the current frame. The teacher receives the current view and up to seven sampled historical views. Its final aggregator output contributes frame and global features, camera and register tokens are removed, and the resulting patch targets are resized to the policy's visual grid.
Let $h_{t,j}$ be the policy hidden state at block 27 and $f_{t,j}^{\mathrm{long}}$ the corresponding teacher feature. A two-layer projector $P_{\mathrm{long}}$ maps the policy state to the teacher dimension. The long-horizon loss is:
$$\mathcal L_{\mathrm{Long}}=\frac{1}{N_t}\sum_{j=1}^{N_t}\left[1-\cos(P_{\mathrm{long}}(h_{t,j}),f_{t,j}^{\mathrm{long}})\right].$$
The teacher adds a UV positional embedding with scale 0.1 before resizing the long-horizon target. This gives the student more than appearance consistency: corresponding patches should also carry compatible geometric placement. Teacher features are detached, so the teacher defines a target but is never optimized by the navigation loss.
Long-horizon guidance is complemented by a local temporal path. Before the language model, current visual-encoder tokens $z_{t,j}$ are compared with tokens from up to three preceding frames. The match score combines cosine similarity with a quadratic positional penalty:
$$s_{u,i,j}=\cos(z_{u,i},z_{t,j})-0.05\|q_i-q_j\|_2^2.$$
Only mutual best matches are accepted. At most one quarter of the current grid is retained per past frame, and temporal weights $0.2$, $0.35$, and $0.5$ emphasize more recent views. For matched pairs $\mathcal M_t(j)$, the residual and fused token are:
$$r_{t,j}=\frac{\sum_{(u,i)\in\mathcal M_t(j)}w_{t-u}(z_{u,i}-z_{t,j})}{\sum_{(u,i)\in\mathcal M_t(j)}w_{t-u}},\qquad \widetilde z_{t,j}=z_{t,j}+g\,r_{t,j}.$$
The scalar gate is $g=0.2\operatorname{sigmoid}(a)$, initialized near 0.05. This bounded gate protects the current observation from aggressive changes caused by occasional bad matches. The discrete selection has no gradient, but gradients flow through the selected feature values. The fused tokens preserve the grid shape, so the vision-language model sees the same interface it already expects.
To teach the fusion path what information is useful, the teacher compares a local temporal window with the current frame alone. After independent normalization, subtraction, and another normalization, the target is:
$$\Delta f_{t,j}=\mathcal N\!\left(\mathcal N(f_{t,j}^{\mathrm{local}})-\mathcal N(f_{t,j}^{\mathrm{single}})\right).$$
A second projector maps the pre-gate residual to this difference:
$$\mathcal L_{\mathrm{Local}}=\frac{1}{|\mathcal J_t|}\sum_{j\in\mathcal J_t}\left[1-\cos(P_{\mathrm{local}}(r_{t,j}),\Delta f_{t,j})\right].$$
Only patches with valid matches belong to $\mathcal J_t$. The target describes the change induced by nearby views, not the full teacher feature. This prevents the residual branch from duplicating the long-horizon alignment task. Long guidance shapes the current policy state; local guidance shapes the residual that temporal fusion actually adds.
The Complete Training Objective
The spatial term combines long and local supervision, and the full objective adds all auxiliary signals to action-token cross entropy:
$$\mathcal L_{\mathrm{Spatial}}=\lambda_{\mathrm{long}}\mathcal L_{\mathrm{Long}}+\lambda_{\mathrm{local}}\mathcal L_{\mathrm{Local}},$$
$$\mathcal L=\mathcal L_{\mathrm{Action}}+\eta(s)\left[\lambda_{\mathrm{sub}}\mathcal L_{\mathrm{Subtask}}+\lambda_{\mathrm{lm}}\mathcal L_{\mathrm{Landmark}}+\mathcal L_{\mathrm{Spatial}}\right].$$
The weights are $(\lambda_{\mathrm{sub}},\lambda_{\mathrm{lm}},\lambda_{\mathrm{long}},\lambda_{\mathrm{local}})=(0.01,0.01,0.5,0.1)$, with a 3 percent auxiliary warmup. The large long-horizon weight reflects dense patch-level supervision across the entire image grid. Landmark and subtask objectives are concentrated on selected token groups, so smaller coefficients are sufficient.
The schedule function $\eta(s)$ introduces guidance gradually. At the beginning of Stage 3, the LoRA adapters have not yet adapted the policy to the annotated expert data. A full auxiliary loss at initialization could pull the representation toward teacher features before the action path is stable. Warmup lets action supervision establish the task while the auxiliary terms become increasingly influential.
The Landmark-Centric Data Engine
Guided-R2R and Guided-RxR preserve the original observations, instructions, and action targets while adding step-aligned subtask spans and landmark masks. For R2R, FGR2R supplies subtask boundaries. These boundaries are matched to continuous trajectories through recorded poses, and nondecreasing anchor-arrival indices define frame-level subtask assignments. Exports with missing poses, ambiguous matches, or incompatible path topology are rejected.
RxR requires a larger construction pipeline. The English training instructions are decomposed by gpt-oss-120b into subtasks, landmarks, and completion conditions. Word timestamps and guide poses provide coarse subtask intervals. GLM-4.5V uses trajectory overviews and frames near boundaries to refine those intervals. Within each aligned clip, it selects explicitly mentioned landmarks or visually supported structures implied by the action.
GroundingDINO verifies candidate boxes, and GLM-4.5V chooses anchor frames. SAM2 propagates accepted box prompts inside the subtask interval. Masks then pass area, detection-consistency, and temporal-support filters. This cascade does not guarantee perfect labels, which is why unreliable annotations disable only their associated loss. The original trajectory remains part of action training even when a mask or span is rejected.
Figure 3: Subtask decomposition, temporal alignment, landmark selection, box verification, and mask propagation produce training annotations while preserving the original navigation data.
Offline annotation may inspect an expert trajectory from beginning to end, but the policy does not receive future observations. At each decision, it sees only current and past context. The teacher is also restricted to available observations. This distinction prevents the convenience of offline segmentation from becoming an invalid information leak during navigation.
Three-Stage Training and Inference
Stage 1 fully fine-tunes Qwen3-VL-8B-Instruct on about 0.53 million R2R expert samples, 1.90 million RxR expert samples, and 0.79 million visual instruction-tuning examples from LLaVA-Video and VSI. Stage 2 continues full-parameter training on 2.97 million DAgger samples mixed with VQA data. Stage 3 applies rank-32 LoRA adapters to annotated R2R and RxR expert data without DAgger or VQA.
Each stage uses one epoch and a global batch size of 256. The optimizer is AdamW with cosine decay, 3 percent warmup, weight decay 0.01, BF16, and DeepSpeed ZeRO-2. Stage 1 language-model peak learning rate is $2\times10^{-5}$ and Stage 2 uses $1\times10^{-5}$. The vision tower and multimodal projector use $5\times10^{-6}$ in both full-tuning stages. Stage 3 uses LoRA scaling factor 16 and a peak learning rate of $2\times10^{-4}$.
The Base Model is not an unadapted foundation model. It is the same Stage-2 checkpoint trained for one epoch with the same rank-32 LoRA budget on the original R2R and RxR data, without the guidance objectives. This controls for adapter capacity and Stage-3 exposure when measuring the contribution of guidance.
Evaluation uses greedy decoding, up to eight context images, and action chunks of up to four primitives. A forward action moves 0.25 meters and a turn rotates 15 degrees. Episodes allow at most 500 primitive actions. At deployment, the geometry teacher and auxiliary projectors are removed; temporal fusion and direct action prediction remain. The resulting policy has 8.77 billion parameters but avoids the online cost of VGGT or generated reasoning.
Navigation Results
On R2R-CE val-unseen, GuidedNav reaches a navigation error of 3.83 meters, 71.5 percent oracle success, 65.4 percent success, and 59.1 SPL. On RxR-CE val-unseen, it reports 4.40 meters, 64.9 percent success, 54.3 SPL, and 69.2 nDTW. Relative to the strongest prior result for each metric, success improves by 8.1 percent on R2R and 13.1 percent on RxR, corresponding to absolute gains of 4.9 and 7.5 percentage points.
| Method | R2R NE | R2R OSR | R2R SR | R2R SPL | RxR NE | RxR SR | RxR SPL | RxR nDTW |
|---|---|---|---|---|---|---|---|---|
| JanusVLN | 4.78 | 65.2 | 60.5 | 56.8 | 6.06 | 56.2 | 47.5 | 62.1 |
| NavFoM | 5.01 | 64.9 | 56.2 | 51.2 | 5.51 | 57.4 | 49.4 | 60.2 |
| StreamVLN | 4.98 | 64.2 | 56.9 | 51.9 | 6.22 | 52.9 | 46.0 | 61.9 |
| GuidedNav | 3.83 | 71.5 | 65.4 | 59.1 | 4.40 | 64.9 | 54.3 | 69.2 |
The joint gain in success and SPL matters because endpoint accuracy alone can hide wasteful detours. The RxR nDTW gain of 10.5 percent indicates that GuidedNav follows the instructed route more closely, not merely that it eventually stops near the target. This pattern is consistent with guidance that keeps the current subtask and its spatial evidence connected during navigation.
Performance by route length gives a sharper interpretation. In the two shorter groups, success changes little and SPL is slightly lower. GuidedNav improves both metrics in the 8.43 to 10.43 meter group and in routes longer than 10.43 meters. The method is therefore most useful when a policy must carry instruction progress across many decisions. Short routes offer less opportunity for accumulated drift, and their small SPL decline suggests that guidance is not a universal efficiency gain.
Runtime
| Method | Parameters (B) | Latency (ms/step) | Throughput (steps/s) |
|---|---|---|---|
| StreamVLN | 8.03 | 87 | 11.4 |
| JanusVLN | 9.31 | 864 | 1.1 |
| MonoDream | 2.00 | 211 | 4.7 |
| GuidedNav | 8.77 | 81 | 12.3 |
Measured on one RTX 5090 and amortized over executed primitive actions, GuidedNav takes 81 milliseconds per step. It is 10.7 times faster per step than JanusVLN and 2.6 times faster than MonoDream, while reducing latency by 6.9 percent relative to StreamVLN. The result is architectural rather than a claim that the model itself is smaller. Spatial learning has already happened in training, so another geometry encoder is not needed during each decision.
Ablation and Representation Analysis
Against the matched Base Model, subtask guidance raises success by 3.6 points, landmark guidance by 3.3, and spatial guidance by 3.1. The two attention objectives produce the largest oracle-success gains, suggesting that they improve access to evidence near the goal. Spatial guidance also raises SPL by 1.1 points. Combining all three reaches 65.4 success and 59.1 SPL, improving over Base by 6.1 and 3.0 points and exceeding the individual modules on both metrics.
| Configuration | NE | OSR | SR | SPL |
|---|---|---|---|---|
| Base Model | 4.87 | 63.8 | 59.3 | 56.1 |
| Base + subtask attention | 4.34 | 72.3 | 62.9 | 55.2 |
| Base + landmark attention | 4.38 | 72.1 | 62.6 | 55.8 |
| Base + spatial guidance | 4.33 | 68.5 | 62.4 | 57.2 |
| GuidedNav | 3.83 | 71.5 | 65.4 | 59.1 |
The progressive spatial ablation starts at 61.8 success and 56.6 SPL with long-horizon guidance. Adding gated fusion changes these to 62.0 and 56.7. Adding local residual supervision reaches 62.4 and 57.2. The larger SPL improvement from local supervision suggests that the content of the fused residual is more important than merely having a temporal pathway.
Attention localization is measured with subtask AUPRC and AUROC, landmark Pointing Game, and patch-level energy-based Pointing Game. Spatial information is measured by fitting a depth probe to frozen policy features. The probe and visualization decoder are analysis tools; neither contributes to action prediction. GuidedNav improves subtask AUPRC from 34.5 to 56.3 percent, landmark Pointing Game from 15.2 to 49.1 percent, AbsRel from 0.25 to 0.19, RMSE from 0.82 to 0.64 meters, and near-far ranking accuracy from 85.9 to 91.7 percent.
| Model | Subtask AUPRC | Subtask AUROC | Landmark PG | Landmark EBPG | AbsRel | RMSE (m) | Rank Acc. |
|---|---|---|---|---|---|---|---|
| Base Model | 34.5 | 53.2 | 15.2 | 13.4 | 0.25 | 0.82 | 85.9 |
| GuidedNav | 56.3 | 76.1 | 49.1 | 42.6 | 0.19 | 0.64 | 91.7 |
The simultaneous rise in pointing and energy-based pointing is informative. Pointing Game checks whether the peak location falls inside a landmark; energy-based pointing measures how much probability mass lies on the mask. A model can improve the first by moving one peak while leaving most attention elsewhere. GuidedNav improves both, showing broader support for the accepted region. The subtask AUROC gain likewise indicates better separation of active spans from completed and upcoming instructions.
Figure 5: Instruction attention at sampled decisions. Green underlines identify the active subtask, while darker blue denotes stronger native attention at the pre-action position.
Figure 6: Reference landmark masks and pre-action attention for stairs and chairs. GuidedNav covers more of each accepted region, while the maps remain relative attention rather than causal attribution.
Real-Robot Evidence
The authors deploy GuidedNav on an AgiBot X2 humanoid with a forward-facing RGB camera. An off-board RTX 4090 server sends predicted actions to the robot, and no real-world fine-tuning is used. In the main demonstration, the robot passes orange chairs and stops at a coffee machine. Native attention shifts from the intermediate landmark to the destination as the instruction progresses.
This is encouraging evidence that the learned language-landmark relation survives a change from simulated to real images. It is also limited evidence. The paper reports qualitative examples rather than a protocol with repeated trials, success rate, collision rate, varied lighting, or communication-delay analysis. The depth shown in the deployment figure comes from an analysis probe and is not an input or a claimed online perception product.
Limitations
The evaluation is concentrated on R2R-CE and RxR-CE val-unseen with monocular RGB input. The method has not established robustness in crowded spaces, outdoor terrain, multi-floor buildings, or deployments that require richer sensing. Real-world trials demonstrate feasibility but do not estimate a failure distribution.
The authors explicitly caution that the attention visualizations measure relative routing within instruction tokens or image patches. They do not establish the causal contribution of an individual token to an action. Better localization scores are consistent with better decisions, but they are not a complete explanation of the policy.
The guidance labels depend on a chain of generated decompositions, detector proposals, and propagated masks. Errors from gpt-oss-120b, GLM-4.5V, GroundingDINO, or SAM2 can reduce supervision quality. Missing annotations disable individual losses, but the paper does not report annotation acceptance distributions or sweep the performance sensitivity to each upstream model. A large missing-label rate could turn GuidedNav into a weaker version of the base policy.
The route-length analysis shows no SPL improvement on shorter paths. This is not a small implementation detail; it defines where the method should be applied. GuidedNav is most justified when long instruction sequences create progress drift. The 8.77 billion parameter policy also remains expensive, and the paper does not test full onboard execution, long-duration operation, or recovery after real-world failure.
Finally, reproducibility is currently incomplete. The public repository contains only documentation and media at the inspected commit. Checkpoints, datasets, training code, and inference code are not available yet, so independent teams cannot reproduce the three-stage recipe or the annotation pipeline.
Conclusion
GuidedNav treats the pre-action position as the place where linguistic progress, visual reference, and spatial context must meet. It supervises native attention without adding a dedicated reasoning branch, distills long-horizon geometry without running the teacher online, and uses local temporal supervision to make history-conditioned residuals informative. The full system improves success, route adherence, and representation quality while retaining direct low-latency action prediction.
The broader lesson is that auxiliary supervision does not have to remain as an inference-time explanation. A model can use rich labels and a large geometry teacher during training to shape the state that controls action, then deploy only the compressed consequence of that supervision. The next step is not another conceptual branch, but stronger evidence: public code and weights, larger real-world evaluation, explicit annotation-quality studies, and tests of whether the learned spatial representation transfers beyond the two benchmark environments.
Golden line: GuidedNav lets the geometry teacher leave after class while the policy keeps the spatial sense it needs for the next step.
