One Scan is Enough: Demonstration-Free Adaptation for Language-Guided Navigation

Authors: Zhuoyuan Yu, Yuxing Long, Du Wang, Junyang Wang, Hongwei Fan, and Hao Dong. Affiliations: CFCS, Peking University; PrimeBot. Paper: arXiv:2610.33794. Code status: No public code is provided.

One-sentence summary

The paper compiles one handheld scan into language-guided navigation tasks and uses a geometry-derived Pareto utility for reinforcement post-training, allowing a continuous waypoint policy to adapt to five real environments without target-scene demonstrations.

Why scene adaptation remains difficult

A language-guided mobile robot must turn a request such as “go to the printer along the corridor” or “find the white mug in the kitchen” into a continuous sequence of safe movements. Vision-language-action models have made substantial progress on public navigation benchmarks, but those benchmarks do not remove the deployment problem. Every new home, office, or teaching building changes the visual appearance, traversable geometry, object placement, and the way users describe destinations.

The standard remedy is to teleoperate a set of fresh demonstrations in the target building and fine-tune the policy on them. This procedure supplies task-relevant trajectories and implicitly captures local execution behavior, but it does not scale. A deployment team would need to repeat the process for every room, route, and instruction family in the long tail of real spaces.

The authors therefore ask a more restrictive question: can a robot adapt from one passive scan when no target-scene task demonstration is available? A scan is attractive because it is task-independent. One operator can walk through a space once, and the resulting observations can in principle support many navigation requests. The challenge is to convert that raw reconstruction into executable supervision rather than merely a photorealistic renderer.

Prior real-to-sim-to-real systems show that 3D Gaussian Splatting can reconstruct a real environment and render robot observations with a much smaller visual gap than conventional simulators. That capability solves part of the problem, but not the language or action problem. A reconstructable scene still needs start-goal pairs, admissible paths, grounded instructions, waypoint labels, and a training signal that survives transfer to the physical robot.

The paper’s central move is to make the scan do all of these jobs. The scan is represented as a Gaussian appearance layer and a queryable bird’s-eye-view geometric layer. The first layer renders first-person observations, while the second supports free-space sampling, clearance-aware planning, safety checks, and goal verification. Scan2Instruction uses those layers to compile tasks, and PRPT-KS uses the same geometry to post-train a navigation VLA.

Problem formulation and the RealNav policy

Let $S$ be a single passive scan of the target environment and $i$ be a natural-language instruction supplied at test time. A policy $\pi_\theta(a_t\mid o_{1:t},i)$ maps observation history and language to an action. For a rollout $\tau=(o_{1:T},a_{1:T})$, the binary outcome $R(\tau,i)\in\{0,1\}$ indicates task success. Adaptation starts from parameters $\theta_0$ pretrained without target-scene access and may use the scan, but no human demonstration in that scene:

$$\pi_{\theta^\star}=\arg\max_\theta \mathbb{E}_{i\sim I}\mathbb{E}_{\tau\sim\pi_\theta}[R(\tau,i)],\qquad \theta=A(\theta_0,S).$$

This constraint rules out several familiar mechanisms. The method cannot learn a target-scene critic from many physical trials, cannot fit a separate reward model, and cannot recover preferences from human labels. It must extract useful objectives directly from the scanned environment. The paper’s contribution is valuable precisely because it turns that constraint into a concrete data compiler and training loop rather than treating it as an abstract aspiration.

RealNav is the continuous-trajectory VLA used in the experiments. Its language and visual inputs pass through a Qwen3-VL chat backbone. Dedicated navigation tokens are appended after the user turn, absorb cross-modal context through self-attention, and expose final hidden states to a lightweight action head:

$$\mu_\theta(s_t)=f_{\mathrm{head}}(h_{\mathrm{nav}}(s_t;\theta_{\mathrm{vlm}}))\in\mathbb{R}^{3K}.$$

The model predicts $K=8$ future local waypoints, producing 24 values: planar displacement $x_k$, planar displacement $y_k$, and heading change $\theta_k$ for each waypoint. The robot infers stopping when consecutive executed waypoints differ by less than preset position and heading thresholds. A stop is correct only when it occurs inside the evaluation radius.

Before adaptation, RealNav is trained by behavior cloning on R2R-CE and RxR-CE. Reference trajectories are converted to local waypoint targets, and the deterministic action head minimizes mean squared error. During post-training, the head is treated as the mean of a Gaussian policy $\pi_\theta(a_t\mid s_t)=\mathcal{N}(\mu_\theta(s_t),\Sigma)$, with isotropic covariance in normalized waypoint coordinates. This allows the same output representation to support policy-ratio updates and pairwise preference likelihoods.

Scan2Instruction: turning reconstruction into a task generator

The scan comes from a handheld iPhone with RGB-D and LiDAR sensing. Per-frame poses are supplied by ARKit or estimated offline with ORB-SLAM3. The operator only needs to cover the environment; there is no instruction to follow and no expert action to demonstrate. Scan2Instruction then creates a joint scene representation from the posed observations.

The appearance layer is a planar-regularized 3D Gaussian representation reconstructed with LiDAR depth supervision. Oversized projected Gaussians can occlude the scene and blur the first-person view, so the system displaces them far along the viewing ray. The geometric layer is obtained by slicing an obstacle-height band from Gaussian occupancy, thresholding, applying morphological cleanup, and retaining connected free-space components.

Figure 1: Scan2Instruction builds coupled appearance and geometric layers, plans collision-aware routes, and asks a multimodal model to synthesize three instruction families.

The reconstructed map requires one-time manual alignment and traversability confirmation. This is not a cosmetic operator step: every later path, safety margin, stop test, and language grounding decision inherits errors from it. After confirmation, the system computes a continuous clearance field over the eroded planning map $M$. For occupied cells $O$, clearance is the Euclidean distance to the nearest blocked location:

$$c(x)=\min_{x'\in O}\lVert x-x'\rVert_2.$$

Clearance is used in route search and in later utility checks. Unlike a binary graph membership test, it gives the planner a graded measure of how far a pose is from obstacles. This is important because the policy outputs dense local waypoints rather than selecting one graph node at a time.

Training starts are sampled inside $M$, and goals are sampled from the same connected component. If a candidate goal has clearance above a threshold, it is moved to the nearest cell whose clearance is below that threshold, placing the endpoint near an object that language can name. A* then searches between selected start-goal pairs with a clearance-penalized edge cost:

$$\operatorname{cost}(x\to x')=s(x,x')+\frac{w}{\max(d,\epsilon)}m(d),$$ $$m(d)=\begin{cases}d_{\min}/(d+\epsilon),&d<d_{\min}\\1,&d\ge d_{\min}\end{cases}.$$

Here $s$ is the grid-step length, $d$ is obstacle distance, $w$ controls the clearance penalty, $d_{\min}$ is the preferred separation, and $\epsilon$ prevents division by zero. Endpoints and traversed cells must remain inside the eroded map. Safety-constrained Ramer-Douglas-Peucker simplification then removes points only when the resulting chord preserves the minimum clearance. Accepted start-goal pairs are sampled in both directions.

The planner produces motions, but the training set also needs language that actually describes those motions. A pretrained multimodal large language model receives guided route frames, goal views, and a type-specific prompt. It generates three task families: Route following describes actions and landmarks; Object Goal names a destination object without giving the route; Human Demand expresses a functional need without directly naming the target or listing turns.

$$i=\operatorname{MLLM}(\{\tilde o_t\}_{t\in T_\gamma},G;P_\gamma).$$

Guided route frames contain an orange future-path trace, and goal views mark the destination with a red ground circle. These guides are inputs to instruction synthesis only. The policy observations are rendered separately with different camera intrinsics and height, and the visual guides are removed. Consequently, RealNav cannot simply copy a colored path from its input and must learn the relationship between language, scene appearance, and robot motion.

Each compiled episode combines the generated instruction, policy observations, reference poses, local waypoint labels, full planner path, and task-family identifier:

$$e=(i,\{o_t\}_{t=1}^T,\{p_t\}_{t=1}^T,\{a_t^\star\}_{t=1}^T,\tau^\star,\gamma).$$

Evaluation instructions receive an additional manual check. This step reduces the chance that an incorrect generated phrase is mistaken for a policy failure, although the paper does not report the rejection rate or the semantic-error rate of unchecked training instructions.

The five target scenes cover an office, a cottage, a teaching building, and two apartments. They contain 768 square meters in total and require 62 minutes of scanning. Scan2Instruction produces 1,500 training episodes, 500 simulated test episodes, and 200 real-robot trials. The per-scene distribution is shown in Table 1.

Scene	Area (m²)	Scan (min)	Training episodes	Simulation tests	Real trials
Office	318	20	300	100	60
Cottage	155	13	300	100	40
Teaching	130	11	300	100	40
Apartment 1	90	9	300	100	30
Apartment 2	75	9	300	100	30
Total	768	62	1,500	500	200

PRPT-KS: one Pareto utility, three training uses

PRPT-KS begins from the observation that a scanned known scene already exposes much of the information usually learned by expensive reinforcement-learning components. Free-space membership, a reference route, obstacle distance, goal position, and correct stopping are queryable. The method therefore avoids a learned critic, a learned reward model, large-scale exploratory pretraining, and human preference collection.

For a state-action pair, the system computes a three-axis utility vector:

$$U(s_t,a)=(U_{\mathrm{prog}},U_{\mathrm{safe}},U_{\mathrm{stop}}).$$

The axes represent progress along the route, safety with respect to obstacles, and correct stopping near the goal. Pareto dominance in $\mathbb{R}^3$ means no axis is worse and at least one axis is strictly better. The same vector is read three ways: a scalar projection for policy-gradient advantage, a per-axis admissibility condition for planner intervention, and a dominance relation for weighting preference pairs.

Progress is defined through a potential over the route. Let $p_t$ be the robot pose, $\rho_t$ the completion fraction associated with the nearest reference pose, $\delta_t$ the distance to that pose, and $\delta_{\mathrm{dead}}$ a deviation dead zone:

$$\Phi_h(p_t)=R_s f(\rho_t)\exp(-\alpha\bar{\delta}_t),\qquad \bar{\delta}_t=\begin{cases}0,&\delta_t<\delta_{\mathrm{dead}}\\\delta_t,&\text{otherwise}\end{cases}.$$

$R_s$ scales the potential, $f$ increases monotonically with route completion, and $\alpha$ controls how quickly a deviation reduces the score. Movement along the reference route increases the potential, while a pose far from that route receives an exponential penalty. The progress utility is the change between adjacent potentials:

$$U_{\mathrm{prog}}(s_t,a)=R_s\Phi_h(p_{t+1})-\Phi_h(p_t).$$

Safety checks the interleaved execution subset $E(a)$ of predicted waypoints against the eroded planning map $M$. Any executed point outside $M$ incurs a fixed penalty:

$$U_{\mathrm{safe}}(s_t,a)=-\lambda_{\mathrm{obs}}\mathbf{1}[\exists w\in E(a):w\notin M].$$

The stopping term rewards a stop action only when the next pose lies within the success radius of goal $g$. This distinguishes useful termination from arbitrary hesitation or a policy that learns to stand still:

$$U_{\mathrm{stop}}(s_t,a)=R_{\mathrm{succ}}\mathbf{1}[a\text{ is stop}\land\lVert p_{t+1}-g\rVert_2<r_{\mathrm{succ}}].$$

The components are summed into a single-step training signal. Generalized advantage estimation aggregates these signals over time, and peak normalization limits their scale. The policy executes deterministic mean actions, with planner substitution when needed. Every action that is actually executed, including a substituted action, enters the clipped importance-ratio objective:

$$A_t=U_{\mathrm{prog}}+U_{\mathrm{safe}}+U_{\mathrm{stop}},$$ $$L_{\mathrm{PG}}=-\mathbb{E}_t[\min(\rho_t(\theta)\hat A_t,\operatorname{clip}(\rho_t(\theta),1-\epsilon,1+\epsilon)\hat A_t)].$$

The second read-out is a per-step admissibility gate. A model action is accepted only when it is both safe and not too far behind in progress:

$$\operatorname{accept}(a_t^{\mathrm{model}})\iff U_{\mathrm{safe}}\ge0\land U_{\mathrm{prog}}\ge-\eta_A.$$

Safety is not allowed to trade against progress because the two axes are checked separately. A collision-prone action is rejected even if it appears to move closer to the goal, and a safe but strongly regressive action is also rejected. The planner supplies a replacement at the same state. Instead of discarding the rejected action, the method retains it as a negative example.

Figure 2: The geometry-derived Pareto utility drives the policy-gradient update, gates planner intervention, and weights rejected model-planner preference pairs.

The third read-out turns each intervention into a continuous-trajectory preference pair. The planner action is paired with the rejected model action at the same state. The pair receives a high weight when the planner action strictly Pareto-dominates the model action and also stops correctly, a medium weight when it strictly dominates without that extra stopping condition, and a low weight when it is merely rejected by admissibility without strict dominance:

$$w_t\in\{w_{\mathrm{high}},w_{\mathrm{mid}},w_{\mathrm{low}}\}.$$

These weights enter a pairwise preference loss. $\Delta_\theta$ and $\Delta_{\mathrm{ref}}$ are the current and reference log-ratio gaps between the planner and model actions under the same Gaussian parameterization, while $\beta$ is a temperature:

$$L_{\mathrm{pref}}=-\mathbb{E}[w_t\log\sigma(\beta(\Delta_\theta-\Delta_{\mathrm{ref}}))].$$

Three stabilizers keep this loop from drifting. A step-wise expert buffer samples $(s,a^\star)$ transitions along planner reference paths and supplies a behavior-cloning anchor. A short-to-long trajectory curriculum separates horizon growth from policy improvement. The covariance $\Sigma$ follows a non-learned cosine schedule that shrinks action dispersion during training. The unified loss is:

$$L=\lambda_{\mathrm{PG}}L_{\mathrm{PG}}+\lambda_{\mathrm{rep}}L_{\mathrm{rep}}+\lambda_{\mathrm{pref}}L_{\mathrm{pref}}.$$

This design is more specific than “use a planner as a teacher.” The planner appears as a reference route and correction source, but the same closed-form utility also defines the policy-gradient advantage and the relative weight of correction pairs. The utility is therefore a shared interface between geometry, online actions, and offline preference supervision.

Evaluation and empirical evidence

The experiments use an office, a cottage, a teaching building, and two apartments, each reconstructed once into a Gaussian scene and geometric map. The evaluation compares DualMap, a 3D scene-graph mapping baseline, with five recent navigation VLAs: NaVILA, Uni-NaVid, StreamVLN, OmniNav, and CorrectNav. All learning-based systems receive the same RGB stream and share a low-level controller.

Two strong baselines fine-tune CorrectNav or RealNav on 20 teleoperated trajectories per scene. These human-demo models represent the usual target-scene adaptation workflow. The central comparison is between that approach and RealNav post-trained with PRPT-KS, which uses no target-scene task demonstrations.

Table 2 reports average success over the reconstructed scene and the physical robot. RealNav without scene-specific post-training reaches 0.23 in simulation and 0.25 on the robot. Its success is comparable to the external VLAs, which is itself informative: generic navigation pretraining does not resolve the mismatch introduced by a new environment.

Method	Simulation SR	Real-robot SR
DualMap	0.28	0.15
NaVILA	0.20	0.20
Uni-NaVid	0.22	0.20
StreamVLN	0.23	0.18
OmniNav	0.23	0.22
CorrectNav	0.22	0.22
CorrectNav + human demos	0.44	0.48
RealNav	0.23	0.25
RealNav + human demos	0.44	0.49
RealNav + PRPT-KS	0.85	0.85

Human demonstrations nearly double success relative to the pretrained models, reaching 0.48 and 0.49 on the robot. This confirms that target-scene experience is useful. PRPT-KS then raises the average to 0.85 in both simulation and reality, roughly 36 to 37 percentage points above the human-demo baselines.

The per-scene results are consistent rather than driven by one easy environment. RealNav + PRPT-KS obtains simulation/real success of 0.79/0.78 in the office, 0.93/0.95 in the cottage, 0.81/0.83 in the teaching building, 0.85/0.87 in Apartment 1, and 0.87/0.87 in Apartment 2. The close simulation and real values suggest that the post-training signal survives the rendering-to-physical execution gap.

The authors further evaluate regions of the same five buildings that were never scanned, reconstructed, included in synthesized tasks, or visited during RL rollouts. The unscanned test contains 20 real trials per scene, for 100 trials total. RealNav + PRPT-KS retains 0.82 success, while RealNav + human demos reaches 0.41 and CorrectNav + human demos reaches 0.40.

Method	Real SR in unscanned regions
DualMap	0.14
NaVILA	0.18
Uni-NaVid	0.22
StreamVLN	0.22
OmniNav	0.26
CorrectNav	0.29
CorrectNav + human demos	0.40
RealNav	0.22
RealNav + human demos	0.41
RealNav + PRPT-KS	0.82

This experiment weakens the interpretation that the policy merely memorizes the reconstructed mesh. The model still performs well when starts, goals, and intermediate waypoints lie outside the scan. At the same time, the unseen areas belong to the same buildings and share visual style, doors, furniture distribution, and local navigation conventions. The result supports within-building extrapolation, not adaptation to a completely different environment without any scan.

Figure 3: One handheld scan generates three instruction families through Scan2Instruction, which are used by PRPT-KS to adapt RealNav and deploy it on physical robots.

What the ablations reveal

The simulation ablation removes PRPT-KS entirely and then removes individual loss terms. Without post-training, success falls from 0.85 to 0.23, SPL falls from 0.834 to 0.210, and navigation error rises from 1.42 meters to 3.50 meters. The large gap shows that the improvement is not just an artifact of evaluating on tasks generated in the same style as training data.

Removing the behavior-cloning anchor causes the largest single-term degradation: success drops to 0.30 and SPL to 0.293. The policy-gradient and preference objectives can improve task outcomes, but they also allow the waypoint distribution to drift away from executable paths. The expert buffer supplies dense, stable reference actions, which appears necessary for preserving trajectory quality.

Removing the Pareto-utility advantage leaves reference replay, preference supervision, and admissibility gating in place, yet success falls to 0.81. Removing the weighted preference term leaves policy-gradient updates, replay, and planner correction, producing 0.80 success. Both components therefore contribute beyond the safety gate, although less than the dense behavior-cloning anchor.

Variant	SR ↑	SPL ↑	NE (m) ↓	OS ↑
RealNav + PRPT-KS (full)	0.85	0.834	1.42	0.88
RealNav without PRPT-KS	0.23	0.210	3.50	0.27
Without behavior-cloning anchor	0.30	0.293	3.49	0.41
Without Pareto-utility advantage	0.81	0.785	1.31	0.85
Without Pareto-weighted preference	0.80	0.781	1.32	0.83

One result should not be smoothed over: the full model’s terminal navigation error is 1.42 meters, while the variants without the utility advantage or without weighted preference obtain 1.31 and 1.32 meters. The full system is better in success, SPL, and oracle success, but its improvements are not uniform across every metric. The authors explicitly report this inconsistency.

Mechanistically, the three read-outs play different roles. The scalar projection supplies dense policy improvement, the admissibility gate prevents unsafe or strongly regressive actions from being executed, and the weighted preference loss learns from planner corrections. Sharing one utility across these functions reduces the chance that online reward, intervention rules, and preference labels optimize incompatible notions of good navigation.

Limitations and deployment boundaries

The authors state that scanning still requires a human operator. The system removes per-task teleoperation, not all human labor. A person must cover the space, align the map, and confirm traversability. The paper does not quantify operator variability, coverage completeness, or failures caused by reflective surfaces, occlusion, stairs, or very large buildings.

The second author-stated limitation is geometric staleness. If furniture moves, doors change state, or the layout is renovated, the free-space map and clearance field may no longer describe the environment. This can cause unsafe navigation as well as lower success. The authors propose active incremental mapping with divergence detection, but the current pipeline assumes a new scan when the layout changes substantially.

Privacy is part of the paper’s broader-impact discussion. Scans of private spaces may include identifiable people or documents. Assets should be anonymized before release, and the scanner should obtain consent from the space owner. The pipeline can run locally without cloud upload, which is relevant for sensitive deployments, but the paper does not evaluate anonymization errors or the security of the local data path.

An independent engineering limitation is the remaining one-time manual alignment and traversability confirmation. Calling the method demonstration-free is accurate, but not human-free. Misalignment can simultaneously corrupt planning, safety checks, correct stopping, and language grounding. The experiments do not isolate how often this calibration step fails or how the policy behaves when the confirmation is wrong.

Instruction quality is another unresolved dependency. Evaluation instructions are manually checked, which is sensible for measuring policy behavior, but the paper does not report semantic error rates for unchecked training instructions. A multimodal generator may omit a visible landmark, name an object incorrectly, or describe a route that is valid but unnatural. The impact of different instruction generators and correction policies remains unknown.

The empirical scope is five indoor scenes totaling 768 square meters. The 200 real trials are unevenly distributed, with 60 in the office and 30 in each apartment. Success uses a 1.5-meter radius and does not establish performance in outdoor terrain, crowded dynamic spaces, multi-floor navigation, or environments with substantial transparent and reflective geometry.

The hybrid planner also affects how the results should be interpreted. During training and execution, PRPT-KS replaces actions that violate the safety or progress admissibility checks. The final system is therefore a learned VLA plus a geometry-aware corrective controller. This is a reasonable architecture for a previously scanned static space, but it is not a pure end-to-end policy acting without map-based assistance.

Finally, the unseen-region study demonstrates extrapolation beyond the reconstructed area inside the same buildings. It does not demonstrate deployment in a different building without scanning. “One scan is enough” should therefore be read as one reusable scan per target environment, replacing per-task demonstrations, rather than as a claim that no scene prior is needed.

Conclusion

The paper elevates a passive target-scene scan into a first-class training input. Scan2Instruction samples executable routes, compiles three grounded instruction families, and renders policy observations that differ from the guided views used for language generation. PRPT-KS then computes progress, safety, and stopping utilities from the same geometric representation and reuses them for policy gradients, online admissibility, and preference weighting.

The strongest empirical result is the consistent comparison across five environments: the demonstration-free method reaches 0.85 success in scanned regions and 0.82 in unscanned regions, while human-demo adaptation remains near 0.41 to 0.49. The ablation adds a second layer of evidence by showing that reference replay, utility advantage, and weighted preference all matter in the tested configuration.

The most natural next step is active incremental mapping. A robot that detects disagreement between onboard observations and reconstructed geometry could request or perform a local rescan instead of refreshing the whole environment. Coupling language generation to geometric uncertainty would also allow the robot to ask for clarification when a landmark is ambiguous or a route is no longer admissible.

For robotics engineers, the broader lesson is that real-to-sim-to-real should not stop at better rendering. A scan can become a task compiler, a planner prior, and a reward interface when appearance, geometry, language, and action are tied to the same scene. That shared anchor is what turns a short scan into a training asset rather than a static visualization.