AgentVLN: How a 3B VLM Navigates by Calling Perception and Planning Skills

A robot enters an unfamiliar office. The instruction is not "go to coordinate (4.2, 8.7)." It is a sequence of landmarks, turns, spatial relations, and a final object: walk past the cabinets, turn right at the end of the corridor, find the refrigerator and the gray boxes, then stop in front of the white bucket. The robot has no map of this room, no second pass to correct a missed turn, and only the images and depth it can collect while moving.

AgentVLN attacks this problem with a deliberately different division of labor. A compact 3B vision-language model does not try to absorb the entire geometry of a building into its weights. It acts as the high-level brain: it reads the instruction, examines the current visual context, chooses a perception or planning skill, and receives the result of that call. The geometry-heavy work stays in a replaceable skill library, where visual SLAM, mapping, path generation, and robot control can do what they are designed to do.

The result is less a "larger VLM" story than a systems-architecture story. AgentVLN asks whether a small model can navigate better when it is given the right interface to spatial tools, and whether long-horizon navigation becomes easier when the model makes a few semantic decisions instead of predicting every low-level action.

01. Why end-to-end navigation is difficult for a VLM

Vision-and-Language Navigation combines language grounding, visual recognition, spatial reasoning, memory, and control. A useful system must remember that a door was passed two turns ago, recognize that a sofa described in the instruction is the same sofa in the current frame, infer that a table is reachable rather than merely visible, and recover when a person or a chair blocks the planned route. The benchmark makes this harder by evaluating in unseen environments.

Existing systems generally choose one of two compromises. A single-system model maps images and language directly to actions or action chunks. This can look elegant, but it asks the model to learn 3D structure and physical reachability implicitly from trajectory data. In a new room, a plausible-looking action can still be geometrically wrong. A dual-system model separates a language planner from a trajectory generator, but then a new interface problem appears: the planner reasons over 2D visual tokens, while the lower level produces 3D waypoints. The two levels do not necessarily share the same notion of "left," "behind," or "three meters away."

Monocular RGB adds another failure mode. If the instruction says "stop in front of the chair," a VLM can locate the chair in pixels but may not know its depth. If the target is partly hidden, the wrong pixel can project to a waypoint several meters away. Adding a depth model helps, but increases latency, memory use, and prompt length. On a robot that needs local inference, those costs matter.

AgentVLN's response is to keep the VLM's job semantic and compositional, while making the spatial evidence explicit. The model sees an image with candidate paths drawn directly onto it, rather than having to infer a latent 3D map from raw RGB. When the path is clear, it selects a waypoint. When the path disappears behind an obstacle, it asks for a small corrective action. When the destination is visible but its scale is ambiguous, it asks a perception skill a specific question before predicting the target point.

02. The VLM as a brain over a plug-in skill library

AgentVLN uses Qwen2.5-VL-3B as its controller. The VLM receives the instruction, the current egocentric observation, and a temporal history of multimodal context. Instead of producing a continuous velocity command, its action is a structured skill call. At decision time $t_k$, the policy chooses a skill through

$$ c_k \sim \pi_{\theta}(f\mid\mathcal{H}_{t_k},o_{t_k},\mathcal{I}), \qquad f\in\mathcal{F}. $$

Here $\mathcal{H}_{t_k}$ is the history context, $o_{t_k}$ is the current observation, $\mathcal{I}$ is the instruction, and $\mathcal{F}$ is the available skill set. The sampled skill call is $c_k$. This notation hides an important engineering choice: the VLM does not need to emit the full low-level control trajectory. It chooses what kind of operation should happen next and supplies the semantic target or context that operation needs.

The skill library is split into perception-level and planning-level functions. Perception skills answer questions about state, map structure, target visibility, depth, and feasible waypoints. Planning skills consume a selected waypoint or target and execute a closed-loop motion segment. The paper models this as a Partially Observable Semi-Markov Decision Process, or POSMDP. A planning skill can run for several low-level steps before returning control to the VLM. The multi-step transition for a selected skill is

$$ \begin{aligned} P(s_{t_k+\tau},\tau\mid s_{t_k},f_{c_k}) ={}&\sum_{a_1\ldots a_\tau} \left[ \prod_{i=0}^{\tau-1} P(s_{t_k+i+1}\mid s_{t_k+i},a_{t_k+i}) \pi_{f_{c_k}}(a_{t_k+i}) \right]\\ &\cdot \beta_{f_{c_k}}(s_{t_k+\tau}). \end{aligned} $$

The term $\pi_{f_{c_k}}$ is the local controller associated with the chosen skill. The function $\beta$ is a termination condition. If the skill has not finished, the robot keeps executing it. When $\beta$ fires, the VLM receives a fresh observation and begins another decision cycle. This semi-Markov structure makes the architecture practical: a navigation primitive can cover multiple frames and controller updates without forcing the language model to re-decide at the controller frequency.

03. Cross-space mapping: draw the 3D route on the 2D image

The most important interface in AgentVLN is the cross-space representation mapping. The perception layer maintains a global map and proposes feasible 3D waypoints. If those points were passed to the VLM as text coordinates alone, the model would have to mentally relate world coordinates to the current camera pose. Instead, AgentVLN projects the waypoints back into the current image plane, producing pixel-aligned visual prompts. The VLM can now make a semantic choice in the same space where it already recognizes doors, furniture, and signs.

First, a pixel $\mathbf{p}_{img}$ with measured depth $d$ is back-projected into the world using the camera intrinsics $K$, rotation $R_t$, and translation $\mathbf{t}_t$:

$$ \mathbf{P}^{w} =R_t\left(d\cdot K^{-1}\mathbf{p}_{img}\right) +\mathbf{t}_t. $$

As the robot moves, these points update an occupancy map. The planning layer can then generate candidate 3D points $\mathbf{P}_{path}^{w}$. To show a path point in the current view, the transformation is inverted and the point is projected into pixels:

$$ s\cdot\mathbf{p}_{path}^{img} =K R_t^{-1}\left(\mathbf{P}_{path}^{w}-\mathbf{t}_t\right). $$

The scalar $s$ is the depth scale in the camera frame. After projection, the candidate route appears as a marker or set of markers overlaid on the RGB observation. The VLM chooses the marker that best matches the instruction. The chosen pixel is then mapped back to its 3D waypoint, and a planning skill executes the motion. In effect, AgentVLN replaces the question "what are the world coordinates of the next safe position?" with the much easier visual question "which of these visible paths best matches what the person asked for?"

flowchart TD
  A["Instruction + RGB-D observation + 8-frame history"] --> B["Qwen2.5-VL-3B brain"]
  B --> C{"Call a skill"}
  C -->|Perception| D["RTAB-Map / occupancy map / feasible waypoints"]
  D --> E["Project 3D path points into the current image"]
  E --> F["VLM selects a visually grounded waypoint"]
  F --> G["Planning skill executes multi-step motion"]
  C -->|No visible feasible path| H["Context-aware correction: Forward / Left / Right"]
  H --> B
  G --> I{"Target strongly visible?"}
  I -->|No| A
  I -->|Yes| J["QD-PCoT asks for missing depth or geometric evidence"]
  J --> K["Predict target pixel, back-project, dock"]
  K --> B

This interface is also what makes the skill library replaceable. If a deployment uses LiDAR instead of stereo depth, the perception side can change while the VLM keeps the same high-level role. The model is not being asked to relearn its language and visual semantics because the robot gained a new sensor.

04. When the route disappears, take a small corrective step

Long-horizon navigation fails through accumulated error. A robot may miss a doorway by half a meter, turn several degrees too far, or lose the target behind furniture. The global map may still be correct while the current camera view has no usable projection of the planned route. AgentVLN therefore includes a context-driven fine-grained correction strategy.

When no globally feasible waypoint is semantically usable, the VLM can leave the high-level planner and output a short atomic action:

$$ a_t\sim\pi_{\theta}(a\mid\mathcal{H}_t,o_t,\mathcal{I}), \qquad a\in\{\mathrm{Forward},\mathrm{Left},\mathrm{Right}\}. $$

These are not blind long-distance commands. They are local probes: look around a corner, move slightly closer to a junction, or turn away from an occluder. Once the perception layer produces a valid path projection again, the system returns to waypoint selection and semi-Markov skill execution. The correction strategy is the difference between forcing a planner to act on a stale map and allowing it to gather one more piece of evidence before committing.

This behavior is visible in the Habitat examples below. Green points mark feasible route projections generated by perception skills. Red circles show selected or predicted navigation targets. In narrow passages and heavily occluded views, the high-level waypoint may temporarily disappear; the robot then emits fine-grained turns or forward moves until the route becomes visible again.

05. QD-PCoT: ask a depth question before guessing a target

Global navigation gets the robot to the right neighborhood. Local target localization is a different problem: the destination is now somewhere in view, but the robot must identify the exact object and estimate its metric position. A monocular VLM can mistake apparent size for distance. A chair that fills half the image may be close, or it may simply be large. Instructions such as "the chair in front of the sofa" or "stand two meters from the table" make the ambiguity worse.

AgentVLN's Query-Driven Perceptual Chain-of-Thought makes the model expose that uncertainty. If the target is spatially ambiguous, the VLM can generate an intermediate natural-language question such as "How far away is the sofa in front of me?" A perception skill answers with depth or geometric context, and the answer enters the reasoning context. Only then does the model predict the target pixel.

Suppose the target pixel is $\mathbf{p}_{target}^{img}=[u_{target},v_{target},1]^{T}$, and the depth map returns $d_{target}$. The point is back-projected into the camera frame with

$$ \mathbf{P}_{target}^{c} =d_{target}K^{-1}\mathbf{p}_{target}^{img} =\begin{bmatrix} \dfrac{(u_{target}-c_x)d_{target}}{f_x}\\[4pt] \dfrac{(v_{target}-c_y)d_{target}}{f_y}\\[4pt] d_{target} \end{bmatrix}. $$

The focal lengths are $f_x$ and $f_y$, and the principal point is $(c_x,c_y)$. The final step transforms the target from the camera frame to the world frame:

$$ \mathbf{P}_{target}^{w} =R_t\mathbf{P}_{target}^{c} +\mathbf{t}_t. $$

The last-mile planner can now drive to $\mathbf{P}_{target}^{w}$ instead of treating a 2D box center as a physical destination. Crucially, QD-PCoT adds no new model parameters. Its "extra computation" is a call to an existing perception skill and an additional textual reasoning step. The paper argues that this is better than blindly regressing target coordinates from RGB, especially in unstructured rooms where depth and occlusion change the meaning of an instruction.

06. AgentVLN-Instruct and the training setup

The paper also introduces AgentVLN-Instruct, a Habitat-based instruction-tuning dataset designed around the skill interface rather than a single monolithic action policy. Its examples cover four abilities: routing between coarse navigation and fine localization based on target visibility, calling generalizable perception and planning skills, reasoning about target coordinates, and asking questions when spatial evidence is missing. The training data explicitly injects trajectory noise and includes atomic action fallbacks for cases where no visual waypoint is available.

The model starts from Qwen2.5-VL-3B. During instruction tuning, the visual encoder and multimodal projection layer are frozen. This preserves the pretrained visual representation and keeps the trainable surface focused on the language-side skill selection and reasoning behavior. LLaVA-Video-178K is added to the training mixture to reduce the loss of general multimodal capability.

Training or input setting	Value	Why it matters
Base VLM	Qwen2.5-VL-3B	Small enough for local edge inference
Frozen components	Vision encoder and multimodal projector	Keeps visual pretraining intact while tuning the controller behavior
Optimizer	AdamW, batch 128	Large-batch instruction tuning
Learning rate	Cosine schedule, peak $2\times10^{-5}$, warmup 0.03	Gradual adaptation from the pretrained model
Observation	640x480 RGB, 110-degree field of view	Wide egocentric view for navigation landmarks
Temporal context	8 frames	Enough history to maintain route context without overwhelming attention
Training cluster	32 NVIDIA A100 GPUs	Scale of the reported instruction-tuning run

The eight-frame history is not an arbitrary default. The ablation finds that too little history makes the model short-sighted, while too much history dilutes attention with stale visual features. At eight frames, the reported R2R validation result peaks at 67.2% success and 64.7% SPL. With no effective temporal history, success falls to 56.7%.

07. Benchmark results: a smaller controller with stronger geometry

The main evaluation uses the Val-Unseen splits of R2R-CE and RxR-CE in Habitat. The metrics are standard for navigation: Navigation Error (NE) is lower-is-better; Oracle Success (OS), Success Rate (SR), Success weighted by Path Length (SPL), and normalized Dynamic Time Warping (nDTW) are higher-is-better. SPL penalizes unnecessary detours, so a system cannot improve success merely by wandering until it finds the goal.

On R2R-CE, AgentVLN-3B reports 3.88 NE, 73.5 OS, 67.2 SR, and 64.7 SPL. The selected comparisons below preserve the paper's values and show why the result attracted attention: the 3B system leads the listed larger systems on both success and path efficiency.

Method	Parameters	NE ↓	OS ↑	SR ↑	SPL ↑
EfficientVLN	4B	4.18	73.7	64.2	55.9
DualVLN	7.1B	4.05	70.7	64.3	58.5
InternVLA-N1	8.3B	4.83	63.3	58.2	54.0
AgentVLN	3B	3.88	73.5	67.2	64.7

The pattern is similar on the longer and more varied RxR-CE instructions. AgentVLN-3B reports 3.92 NE, 69.5 SR, 61.3 SPL, and 74.6 nDTW. Relative to EfficientVLN-4B, SR rises by 2.5 points and SPL by 7.0 points; the latter is especially important because it says the successful trajectories are more direct.

Method	Parameters	NE ↓	SR ↑	SPL ↑	nDTW ↑
EfficientVLN	4B	3.88	67.0	54.3	68.4
DualVLN	7.1B	4.58	61.4	51.8	70.0
InternVLA-N1	8.3B	5.91	53.5	46.1	65.3
AgentVLN	3B	3.92	69.5	61.3	74.6

These numbers should be read as evidence for a benchmark and implementation, not as a blanket ranking of every navigation system. Success depends on the simulator, observation configuration, training data, and execution stack. The meaningful claim is narrower and more useful: explicit spatial interfaces let a small VLM achieve a stronger accuracy-efficiency trade-off than the listed end-to-end baselines under this evaluation protocol.

08. Ablation: which part actually moves the result?

The incremental ablation is the clearest explanation of the architecture's value. The baseline is an end-to-end small VLM without explicit geometric guidance. Adding VLM-as-Brain and cross-space mapping lifts SR from 38.6% to 59.7% and reduces NE from 6.53 meters to 4.67 meters. This is a large change, and it supports the paper's central hypothesis: the model benefits when it can perform semantic matching over a visible set of feasible waypoints instead of learning 3D reachability from raw pixels.

Adding context-driven fine-grained correction raises SR again to 65.6% and lowers NE to 3.90 meters. The gain comes from handling occlusions and accumulated drift before they become irreversible. Finally, QD-PCoT improves SR to 67.2% and SPL to 64.7%. The final increment is smaller, but it targets the last-meter ambiguity that can leave a robot close to the correct room while failing the precise target condition.

VLM-as-Brain	Fine-grained correction	QD-PCoT	NE ↓	OS ↑	SR ↑	SPL ↑
No	No	No	6.53	48.50	38.60	35.10
Yes	No	No	4.67	65.10	59.70	55.60
Yes	Yes	No	3.90	72.10	65.60	63.40
Yes	Yes	Yes	3.88	73.50	67.20	64.70

09. From Habitat to a Unitree Go2

The real-world prototype uses a Unitree Go2 quadruped with an Intel RealSense D455 camera. RTAB-Map runs local SLAM and incrementally updates a 3D occupancy grid. Camera processing converts the VLM's selected pixel or waypoint into the robot's global frame, while path-generation and control skills turn the result into a smooth obstacle-avoiding trajectory. The VLM and 3D feature fusion are not running in a remote data center for every control cycle; the design is intended for local inference on an edge platform such as Jetson.

The paper shows indoor and outdoor runs. In the office sequence, the Go2 follows a multi-landmark instruction through cabinets, turns right down a passage, and searches for a white bucket near a refrigerator and gray boxes. In the outdoor sequence, it passes a red airplane, crosses between grass regions, and stops in front of a blue billboard. These are meaningful demonstrations because the instruction contains both route structure and visual constraints.

The project README also reports demos on a Unitree G1 humanoid and Go2, and the code release includes training and inference components, a third-party skill library, camera-mount documentation, and dataset-generation tools. That is a broader release than a paper-only artifact, although not yet a complete turnkey stack.

10. What the architecture buys, and what it still leaves open

Strength: explicit geometry stays outside the language model. The VLM chooses among visual prompts produced by geometry-aware tools. This reduces the amount of 3D reasoning the language model must learn implicitly and makes the interface auditable: an engineer can inspect the map, the projected waypoints, and the chosen skill.

Strength: the skill library can change without retraining the brain. A different robot, depth sensor, local planner, or controller can be integrated at the skill layer. The paper describes adapting to LiDAR by replacing the relevant perception skills rather than retraining a multi-billion-parameter model.

Strength: self-correction is local and bounded. The robot does not surrender a long-horizon plan at the first missing waypoint. It takes a small exploration action, refreshes the observation, and resumes when the global path is visible again. This is a sensible compromise between reactivity and deliberate planning.

Open issue: the release is not yet a complete reproduction package. The repository's TODO list still leaves pretrained model checkpoints and installation/environment setup instructions unchecked. The dataset, training code, inference code, deployment skills, and demonstrations are public material, but a new team cannot yet assume that the exact reported model can be downloaded and run end to end.

Open issue: simulation performance is not real-world reliability. R2R-CE and RxR-CE are valuable controlled benchmarks, but they do not cover every lighting condition, moving obstacle, sensor fault, or social interaction in a live building. The real-world evidence in the paper is a set of demonstrations, not a large randomized field trial with confidence intervals.

Open issue: interactive perception adds latency. QD-PCoT avoids extra parameters, but it does not make a perception call free. Asking for depth, receiving it, updating the prompt, and producing a new target coordinate costs time. A deployment must decide when the ambiguity is large enough to justify another round. The paper's edge-deployment argument is about the base model and local execution, not a guarantee that every QD-PCoT trajectory meets a fixed control deadline.

Open issue: modular systems inherit modular failures. If RTAB-Map drifts, if depth is missing on a reflective surface, or if no feasible waypoint is projected, the VLM receives degraded evidence. The corrective action policy can recover from some occlusions, but it cannot manufacture sensor information that the robot never captured.

11. Why this matters beyond one benchmark

AgentVLN points to a practical pattern for embodied AI: use the foundation model for semantic interpretation and coordination, and expose the physical world through explicit, inspectable operations. The VLM remains the place where language, object identity, and task intent meet. Geometry, mapping, and control remain in tools that can be tested and replaced independently.

That pattern is particularly relevant to robots that cannot depend on a remote server. A 3B model is still a substantial workload, but it leaves more room for local execution than a much larger model plus a separate 3D encoder. The cross-space mapping also provides a useful debugging surface: when the robot goes the wrong way, an engineer can ask whether the perception layer proposed the wrong path, whether the projection placed it badly, whether the VLM selected the wrong prompt, or whether the planner failed to execute it.

The most interesting claim is therefore not that a 3B model wins a leaderboard. It is that a VLM can behave more like an agent when its environment is represented through a small set of well-defined skills. The model does not need to hallucinate a perfect 3D world in its hidden state. It needs to know when to look, what to ask, which visible route matches the instruction, and when enough evidence has arrived to move.

The useful abstraction is not "let the VLM do everything." It is "let the VLM decide what to do next, and give it a reliable way to ask the physical world for the missing evidence."

Sources and further reading

Original paper: AgentVLN: Towards Agentic Vision-and-Language Navigation, Zihao Xin, Wentong Li, Yixuan Jiang, Ziyuan Huang, Bin Wang, Piji Li, Jianke Zhu, Jie Qin, and Sheng-Jun Huang. arXiv:2603.17670, accepted at ECCV 2026.

Code: Allenxinn/AgentVLN. Project page: allenxinn.github.io/AgentVLN. Dataset: AgentVLN-Instruct on Hugging Face.

This article is an independent technical adaptation for RobotWorld. Figures are reproduced from the AgentVLN paper; the benchmark numbers and implementation details above are taken from the paper and public project materials.