See-and-Reach: Precise Vision-Language Navigation for UAVs within the Field of View

Paper Metadata

Title: See-and-Reach: Precise Vision-Language Navigation for UAVs within the Field of View (method name: 3DG-VLN)

Authors: Fanfu Xue, En Yu, Yantian Shen, Zhikun Hu, Hongjun Wang, Yang Yang, Xindi Wang, Jiande Sun

Paper: arXiv:2606.20045v1 (2026-06-18)　Code: public at github.com/xuefanfu/3DG-VLN; benchmark on HuggingFace fanfuxue/UAV-VLN-FOV, weights at fanfuxue/3DG-VLN

One-Sentence Summary

The paper carves the final "target already visible but not yet reached" segment out of UAV vision-language navigation, defines it as the UAV-VLN-FOV task with a strict 10-meter success radius, and answers it with 3DG-VLN, a waypoint-prediction framework that consumes high-resolution front and downward views under an online 3D direction cue, lifting success rate from 25.00% to 38.82% on its own benchmark.

Background and Motivation

UAV vision-language navigation asks an aerial agent to fly missions described in natural language while observing the world through egocentric cameras. Compared with ground VLN, aerial space is more expansive and less structured, so success requires multimodal grounding, 3D spatial awareness, and dynamic motion planning under continuous control to converge at once. Despite progress on instruction design, datasets, and model frameworks, terminal reaching precision remains a capability that holistic evaluation keeps hiding.

Most prior UAV-VLN work formulates the task as a monolithic search-and-reach process: discover the target somewhere in a large environment, then approach it. The authors argue this formulation blurs a fundamental divergence. The search stage rewards high-level scene understanding and long-horizon discovery, while the reaching stage, which they name the see-and-reach stage, demands a completely different skill set: fine-grained visual localization and 3D trajectory control. When both are scored by one metric, terminal precision failures get masked by exploration errors.

The divergence becomes lethal in high-stakes missions. The paper's example is post-disaster relief: once a specific emergency supply station is visually identified, the mission hinges entirely on whether the UAV can approach closely enough for safe payload delivery. Current UAV-VLN benchmarks adopt a permissive 20-meter success radius, yet a 20-meter deviation can drop critical supplies into inaccessible terrain. The authors therefore isolate the reaching process and tighten the success radius to 10 meters.

Under this target-visible setting, two concrete deficiencies of existing methods surface. First, methods designed for long-range search use low-resolution observations that suffice for scene-level exploration but cannot capture fine-grained geometric detail, discriminate targets from visually similar distractors, or perceive nearby obstacles, all of which waypoint-level planning and terminal maneuvering require. Second, direction-guided methods rely on a static cue computed at trajectory onset; as the UAV moves, accumulated spatial drift misaligns the linguistic instruction with the current egocentric perspective.

In response, the paper formalizes UAV-VLN-FOV: the target is visible within the field of view at the initial pose, and the agent must translate linguistic instruction plus egocentric visual evidence directly into executable 3D waypoint plans, while surviving radical scale changes, partial occlusion, transient target loss from the field of view, and reactive obstacle avoidance. The accompanying contributions are a high-resolution benchmark (2,717 trajectories, 31,878 image-instruction-waypoint tuples, 14 rural/town/urban scenes, 89 target classes) and the 3DG-VLN framework.

Fig. 1 (paper Fig. 1): UAV-VLN stresses the holistic search-and-reach process, while UAV-VLN-FOV isolates the see-and-reach stage; the instruction examples ask the UAV to reach a yellow car or a yellow tank beside an intersection.

Task Formalization and Preliminaries

The state space is a subset of $\mathcal{S}\subset\text{SE}(3)$; the pose at time $t$ is $\mathbf{P}_{t}=(\mathbf{p}_{t},\mathbf{q}_{t})$ with global position $\mathbf{p}_{t}\in\mathbb{R}^{3}$ and orientation $\mathbf{q}_{t}\in\mathbb{S}^{3}$. Each step yields an observation tuple $\mathbf{X}_{t}=\{\mathcal{I},\mathcal{V}_{t},\mathcal{D}_{t}\}$: $\mathcal{I}$ is the high-level instruction; $\mathcal{V}_{t}=\{\mathbf{R}^{front}_{t},\mathbf{R}^{down}_{t}\}$ holds high-resolution front and downward egocentric views with $\mathbf{R}\in\mathbb{R}^{H\times W\times 3}$, instantiated at 1024x1024; and $\mathcal{D}_{t}\in\{\textit{front, front-left, front-right, down}\}$ is a coarse 3D direction cue serving as a semantically explicit spatial prior.

The action is a short-horizon waypoint sequence $\mathcal{W}_{t}=\{\mathbf{w}_{t+k}\}_{k=1}^{K}$ with $K=5$. Each waypoint $\mathbf{w}_{t+k}=(\Delta x_{t+k},\Delta y_{t+k},\Delta z_{t+k})$ encodes forward, rightward, and downward displacement in the UAV-centric body frame, which grants translation and rotation invariance and lets the flight controller execute the output directly. The policy is a deterministic map from observation history to action:

$$\mathcal{W}_{t}=\pi(\mathcal{I},\mathcal{V}_{\leq t},\mathcal{D}_{t};\theta)$$

Evaluation keeps the standard SR, OSR, NE, and SPL metrics but tightens the success criterion. With terminal position $\mathbf{p}_{T}$, target position $\mathbf{p}_{tar}$, and success radius $\delta=10$ meters (20 meters in prior work), a trajectory succeeds when:

$$S=\mathbb{I}(\|\mathbf{p}_{T}-\mathbf{p}_{tar}\|_{2}\leq\delta)$$

This seemingly small change is the fulcrum of the paper: it separates "flying into the target region" from "reaching the target vicinity", letting the benchmark diagnose terminal control fidelity instead of rewarding regional proximity.

Method: 3DG-VLN in Detail

The framework splits into a training line and an inference line. Training supervised-fine-tunes Qwen2.5-VL-7B (LoRA adapters) on the new benchmark so it regresses smooth waypoints from instruction plus dual views. Inference adds two external modules: DeepSeek-V3.2 parses the target object and its intrinsic attributes from the instruction, while an open-set detector (Grounding DINO) plus an online 3D direction updating module keep the direction cue fresh during closed-loop flight.

High-resolution dual views. Unlike the 256x256 inputs common in long-range UAV-VLN, 3DG-VLN processes 1024x1024 front and downward images in a resolution-adaptive manner without traditional downsampling. The front view carries target morphology and local spatial structure; the downward view covers the area directly beneath the airframe, exactly where the front camera loses the target during final approach. The ablation is blunt: the low-resolution variant (LR) reaches only 23.03% SR on Test, 15.79 points below the full model, so fine-grained visual evidence is a requirement, not a luxury.

Geometry of the direction cue. The body frame axes $x,y,z$ correspond to front, right, and down. Given the UAV pose and target position, the target displacement is first rotated from the global frame into the body frame:

$$\mathbf{d}_{t}=\mathbf{R}(\mathbf{q}_{t})^{\top}(\mathbf{p}_{tar}-\mathbf{p}_{t}),\quad \alpha_{t}=\arctan2(d_{y},d_{x})$$

where $\mathbf{R}(\mathbf{q}_{t})$ maps body-frame vectors to the global frame and its transpose performs the inverse. Yaw angle $\alpha_{t}$ inside a +/-15 degree frontal sector is labeled front, with one sector on each side giving front-left and front-right; in elevation, the cue becomes down when:

$$d_{z}/\|\mathbf{d}_{t}\|>\cos\phi,\quad \gamma_{front}=15^{\circ},\ \phi=\mathrm{FOV}/4,\ \mathrm{FOV}=90^{\circ}$$

So the down threshold is a quarter of the field of view, and together with the 15-degree frontal half-width this compresses continuous direction into four semantic bins. These bins form the cue $\mathcal{D}_{t}$ injected into the prompt: not a precise angle, but a stable, verbalizable spatial prior.

Fig. 2 (paper Fig. 3): the four direction bins in the body frame; front-left, front, and front-right tile the frontal sector while down points at the region covered by the downward camera.

Waypoint supervision in the body frame. Training labels are built with the same transform. For a future step $t+k$, the global displacement is rotated into the body frame:

$$\mathbf{w}_{t+k}=\mathbf{R}(\mathbf{q}_{t})^{\top}(\mathbf{p}_{t+k}-\mathbf{p}_{t})$$

At inference the controller inverts the transform, $\mathbf{p}_{t+k}=\mathbf{p}_{t}+\mathbf{R}(\mathbf{q}_{t})\mathbf{w}_{t+k}$, to recover global waypoints. Body-frame supervision means identical visual-linguistic evidence maps to the same output distribution regardless of global pose: the model learns "how to fly relative to me" instead of memorizing coordinates.

Supervised fine-tuning objective. Waypoints are emitted as autoregressive text tokens with the standard next-token log-likelihood:

$$\mathcal{L}_{SFT}=-\frac{1}{N}\sum_{t}\frac{1}{L_{t}}\sum_{\ell}\log p_{\theta}(s_{t,\ell}\mid \mathbf{X}_{t}, s_{t,<\ell})$$

where $s_{t,\ell}$ is token $\ell$ of the output sequence at step $t$ and $L_{t}$ its length. Implementation uses LoRA-adapted Qwen2.5-VL-7B trained for 2 epochs on a single A100-40GB with batch size 2, gradient accumulation 8, learning rate 1e-4, and warmup ratio 0.1; in the released repo, travel_util.py loads the adapters via PeftModel.from_pretrained, matching the paper.

Online 3D direction updating. This is the inference-side core. At every step, DeepSeek-V3.2 extracts the target object and attributes from the instruction, then Grounding DINO detects on both views. A view counts as valid only if it returns exactly one box: multiple boxes mean ambiguity, zero boxes mean the target is absent. When both views are valid the downward view wins, because during approach the target mostly sits below the airframe. The box center pixel is then back-projected into a body-frame ray:

$$f=\frac{W}{2\tan(\mathrm{FOV}/2)},\quad \mathbf{r}'_{cam}=\left(\frac{u-c_{x}}{f},\frac{v-c_{y}}{f},1\right),\quad \psi=\arctan2(r_{y},r_{x})$$

Pixel coordinates are unprojected through focal length $f$ and principal point $(c_x,c_y)$ into the camera frame, then permuted into the body frame (the repo's CameraModel uses exactly R_cam_to_body = [[0,0,1],[1,0,0],[0,1,0]]); the ray yaw $\psi$ is discretized by the +/-15 degree rule into one of four bins, producing the new $\mathcal{D}_{t}$. If no view is valid this step, the previous cue $\mathcal{D}_{t-1}$ is kept so the prompt never loses its spatial prior during occlusion. The code mirrors the paper line by line: classify_direction implements the +/-15 degree bins, len(boxes)==1 the single-detection validity test, and camera==1 the downward-view priority.

Closed-loop execution and stopping. The model emits $K=5$ waypoints per call and the controller executes them in order while the observe-detect-update-predict loop keeps running. Stopping follows the TravelUAV strategy: an open-set detector plus depth estimate terminates the episode when depth divided by 2.55 falls below a threshold, implemented in eval_qwenvl_direction.py.

flowchart TD
  A["Dual views front+down at 1024x1024"] --> B["Grounding DINO open-set detection"]
  B --> C{"Any view with exactly one box"}
  C -- no --> D["Keep previous direction cue"]
  C -- yes --> E["Prefer downward view when both valid"]
  E --> F["Back-project box center pixel to body ray"]
  F --> G["Discretize yaw into four bins at +/-15 deg"]
  G --> H["Inject cue with instruction and dual views"]
  D --> H
  H --> I["LoRA-tuned Qwen2.5-VL-7B predicts 5 body waypoints"]
  I --> J["Controller executes waypoints via inverse rotation"]
  J --> K{"Depth-based stop rule triggered"}
  K -- no --> A
  K -- yes --> L["End and judge success at 10 m radius"]

The diagram makes the loop auditable: the direction cue is not a constant fixed at training or episode start but a live variable re-estimated from current observations, with a fallback to the previous cue whenever detection is unusable.

Fig. 3 (paper Fig. 2): training fine-tunes Qwen2.5-VL on the new benchmark to predict smooth waypoints; inference adds DeepSeek-V3.2 for target parsing and the online direction updating module for guidance.

Experiments

The benchmark deserves its own reading: 2,717 trajectories split into 2,228 for training, 152 for Test, 164 for Test UO (unseen objects), and 173 for Test US (unseen scenes). Although the target is visible from the start, the longest trajectory still spans 138 meters, and instructions average only 12 words, shorter and closer to real verbal commands than prior UAV-VLN datasets. The length histogram is a right-skewed unimodal curve: "within field of view" does not mean "close".

Fig. 4 (paper Fig. 4b): trajectory length distribution, peaking around 30-50 m with a maximum of 138 m; the target-visible task still involves long-horizon maneuvering.

Main results appear in Table 1. Random and Fixed baselines score near-zero SR on all three splits, and untuned Qwen2.5-VL-7B manages only sporadic oracle success, confirming that a visible target does not make the task easy: the vast, unstructured 3D motion space is hard by itself. Against the current SOTA TravelUAV, 3DG-VLN raises SR on Test from 25.00% to 38.82% (+13.82), OSR from 30.26% to 55.92%, SPL from 17.89% to 27.70%, and cuts NE from 34.03 m to 24.41 m; gains persist on the unseen-object and unseen-scene splits, so the improvement is not overfitting to training scenes.

Method	Test SR	Test OSR	Test NE	Test SPL	UO SR	UO OSR	US SR	US OSR
Random	0.66	0.66	46.34	0.07	0.00	1.22	0.00	1.73
Fixed	0.00	3.29	46.30	0.00	0.00	2.44	0.00	0.00
Qwen2.5-VL-7B	0.00	3.29	42.31	0.00	0.00	3.05	0.58	2.31
TravelUAV	25.00	30.26	34.03	17.89	20.12	24.39	16.76	24.28
3DG-VLN	38.82	55.92	24.41	27.70	28.05	39.63	21.39	35.84

Table 1 (excerpt of paper Table I): main results on the three splits. 3DG-VLN beats TravelUAV on every metric of every split; the OSR jump shows the model not only succeeds more often but also gets genuinely closer when it does not.

The ablations separate the two design choices cleanly (Table 2). The 256x256 low-resolution variant LR scores 23.03% SR on Test; removing the direction cue (3DG-W) gives 25.00%; keeping only the instruction-time direction without online updates (3DG-S) gives 34.87%; the full model reaches 38.82%. Across the three splits the SR margins of the full model over LR are 15.79, 9.76, and 4.05 points, and the OSR margins 19.08, 15.85, and 6.36 points: high resolution and direction guidance each contribute independently, and their gains stack.

Strategy	Test SR	Test OSR	Test NE	Test SPL	UO SR	US SR
LR (256x256 input)	23.03	36.84	35.03	17.05	18.29	17.34
3DG-W (no direction cue)	25.00	44.08	29.86	17.31	22.56	17.34
3DG-S (initial direction only)	34.87	49.34	27.53	24.61	23.17	16.76
3DG-VLN (full)	38.82	55.92	24.41	27.70	28.05	21.39

Table 2 (excerpt of paper Table II): ablating resolution, direction cue, and online updating. Removing any single component degrades performance monotonically, so all three are load-bearing.

When to recompute the cue gets its own experiment (Table 3). Entering the see-and-reach stage with the stale direction from the start of long-range navigation (-O rows) yields 24.34% Test SR for TravelUAV versus 25.00% after stage-level recomputation; for 3DG-VLN the stale cue scores 26.32% while recomputation jumps to 34.87%. Adding continuous online updating on top (34.87% to 38.82%) moves in the same direction: the cue must track current observations. Visualizations agree, with 3DG-VLN shifting the cue from front to front-right as the target drifts in the egocentric view while 3DG-S clings to a static direction and loses alignment.

Strategy	Test SR	Test OSR	Test NE	Test SPL
TravelUAV-O (stale cue)	24.34	26.97	34.81	16.58
TravelUAV (recomputed)	25.00	30.26	34.03	17.89
3DG-VLN-O (stale cue)	26.32	40.79	35.90	19.29
3DG-S (recomputed)	34.87	49.34	27.53	24.61

Table 3 (excerpt of paper Table III): stale versus recomputed direction cues; both stage-level recomputation and closed-loop updating contribute stable gains.

Fig. 5 (paper Fig. 5): navigation in high-fidelity simulation with per-timestep front and downward views; the agent avoids obstacles and reaches the target vicinity on its own.

Real-world trials are qualitative only: under illumination changes and background variation, 3DG-VLN still guides the UAV into close spatial vicinity of the specified target rather than merely the target area, offering preliminary evidence of sim-to-real transfer. No success rates or error numbers are reported for this part, only trajectory screenshots.

Fig. 6 (paper Fig. 7): real-world rollout with per-timestep front and downward observations as the target grows from a distant speck to the center of the frame.

Limitations

Author-stated: dependence on an external open-set detector. The online direction cue is back-projected from Grounding DINO boxes, so direction estimation is only as good as detection; missed, duplicated, or off-center boxes corrupt the cue. The authors list an integrated framework that jointly models visual grounding, spatial alignment, and motion planning as future work.

The single-box validity test is conservative. A view counts only when it returns exactly one box, so frames with a same-class distractor beside the target, or two candidate boxes from the detector, discard that frame's direction information and fall back to the previous cue. In dense urban scenes or multi-target settings this systematically lowers the update rate, and the paper reports no diagnostic such as valid-view hit rate.

Absolute precision remains modest and the real world is unquantified. Even the full model reaches only 38.82% SR under the 10-meter radius on Test, leaving more than six attempts in ten unsuccessful; the real-world section provides screenshots without success rates or errors, so the sim-to-real claim stays preliminary. NE also rises slightly on Test UO when direction guidance is added, which the authors attribute to noisy cues under uncertain unseen-object grounding, an admission that the direction mechanism is still fragile out of distribution.

Conclusion and Outlook

The paper's first contribution is evaluative: cutting the see-and-reach stage out of holistic UAV-VLN and tightening the success radius from 20 to 10 meters makes terminal precision a diagnosable capability for the first time. Under this lens, high-resolution dual views and online direction updating, two choices that look merely engineering-flavored, are shown by ablation to be necessary and stackable.

Methodologically, 3DG-VLN invents no new architecture. It feeds geometric priors (body-frame waypoints, four direction bins, pixel back-projection) as prompt tokens to a LoRA-tuned general VLM and lets external detectors refresh those priors inside the loop. This division of labor, VLM for waypoints and geometry for priors, is a reproducible and auditable route for aerial embodied intelligence; the natural next step is folding grounding, alignment, and planning into one model so the external detector can retire.

Golden Quote

"By isolating the reaching process and advocating for a more stringent 10-meter success criterion, we seek to address a fundamental question in aerial embodied intelligence: how an agent can effectively transform linguistic instruction and visual evidence into the precise 3D maneuvers required for reliably reaching the target vicinity." (Introduction)