See What I Need, Dream What I Want: A Navigation Foundation Model on a Self-Constructed Representation Space

See What I Need, Dream What I Want: A Navigation Foundation Model on a Self-Constructed Representation Space

Authors: Jiawei Chen, Yinxi Wang, Zihan Yang, Meijun Gao, Yaowei Liu, Shuming Zhang, Zecheng Li, Xiangju Mi, Yuxuan Liu, Runzhong Huang, Tianlin Li, Xun Zhang, Zhengyang Wang, Xu Miao, and Tong Yang.

Source: arXiv:2610.31762, version 1, submitted on 22 October 2026. The preprint includes the main paper, appendices, training configuration, and physical-robot evaluation.

Project and code: The project page is public. The GitHub repository currently contains only a README and states that the paper and code are coming soon. This reading therefore distinguishes paper claims from implementation evidence and does not invent function names or source files.

One-Sentence Summary

RepresentNav makes navigation gradients select information from every depth of three frozen vision models, then asks the policy to predict four future states in that same learned space before emitting eight waypoint poses.

Figure 1: The paper's opening overview shows a quadruped tracking a person in a real room, post-hoc semantic, occupancy, annotation, and motion readouts from a dreamed latent, and the compact policy architecture with headline benchmark results.

Why Another Navigation Representation?

A navigation policy cannot act on every pixel equally. Walls constrain motion, floors define traversability, furniture creates bottlenecks, people move independently, and a language instruction determines which of these elements matters now. A representation trained mainly for image-level recognition may preserve object identity while discarding the exact boundary geometry needed to pass a doorway. A representation trained for depth may preserve metric structure while weakening the distinction between a movable chair and a permanent wall.

Modular navigation stacks address this problem by assigning separate modules to mapping, localization, instruction parsing, and planning. The decomposition is legible, but each module has an objective that differs from the final navigation error. End-to-end policies remove that mismatch, yet many of them still consume only the final feature map of a vision-language backbone. The task can update the action head without substantially changing which internal features the policy is allowed to see.

Another common response is to concatenate outputs from several foundation models. This increases information, but treats a shallow edge detector, a deep semantic feature, a depth representation, and a temporal feature as if they occupied the same role. The useful evidence is often sparse and location dependent. A floor region may favor geometry, a furniture boundary may favor semantics, and a moving target may require dynamics. A fixed concatenation leaves the policy to disentangle these sources after they have already been compressed into separate final interfaces.

Future prediction raises a second representational question. World models may forecast text tokens, hand-designed spatial tokens, future pixels, or the latent state of another generative model. If none of these spaces matches the representation used for control, a navigation head must translate between seeing and imagining. The predicted future may look plausible while remaining weakly connected to the policy's actions.

The paper's thesis is that both choices should be made by the navigation objective. The policy should construct the visual space it needs from the internal structure of available priors, and it should dream in that same space. This creates one interface for current perception, future prediction, and waypoint generation rather than three loosely connected interfaces.

Technical Preliminaries

The policy receives a language instruction or goal $L$ and a history of RGB observations $I_{1:t}^{1:N}$ from one or more cameras. It returns eight poses in the current robot frame, $\tau=\{(x_i,y_i,\theta_i)\}_{i=1}^{8}$, where each pose contains planar position and yaw. Point-goal navigation, active tracking, and vision-and-language navigation share this action output. Image and video question answering use the same Qwen3-VL language backbone and are trained as an auxiliary task.

Three frozen models supply complementary visual priors. DINOv3 contributes semantic features, Depth Anything 3 contributes geometric features, and OpenD4RT contributes dynamic features. Freezing avoids retraining these large encoders, but it does not make their use task independent. Three newly initialized transformer layers learn which layer, model, and spatial position to read for the current sample.

The predictive objective follows the joint-embedding predictive architecture family. A predictor estimates future embeddings rather than reconstructing every future pixel. Because the target is generated by the same learnable representation system, an unconstrained predictor can minimize error by collapsing all targets toward a small region. VISReg is therefore applied to the target distribution, while the navigation and prediction losses jointly shape the online representation.

Building the Space the Policy Needs

At position $p$, the $l$-th trainable residual layer issues a query $q_l$ against candidate features from every selected depth of the three encoders. Candidate $f_{l,j}(p)$ is normalized by RMSNorm, scored against the query, and passed through a softmax over candidates. A learned bias is added only to candidate zero, preserving a stable path to the initial or running state.

$$\alpha_{l,j}(p)=\operatorname{softmax}_{j}\left[q_l^{\top}\operatorname{RMSNorm}(f_{l,j}(p))+b_l\mathbf{1}[j=0]\right]$$ $$\widetilde{f}_l(p)=\sum_{j=0}^{J}\alpha_{l,j}(p)f_{l,j}(p)$$

The first equation is the selection mechanism and the second is the selected feature. Read together, they describe a layer that does not commit in advance to one encoder or one depth. Its preference can change across image locations and navigation samples. The model can pull fine geometry from a shallow depth layer in one region while selecting a more semantic feature in another, and the task losses decide whether that mixture helps.

The fused features pass through three residual transformer layers together with a learnable state token. The token attends to the spatial evidence and becomes a per-view summary, called the [CLS] state in the paper. States from multiple cameras are concatenated to form the target for future prediction. This bottleneck is important: the full intermediate evidence can remain spatially detailed, while the language model and planner receive a compact vector per view.

Figure 2: The real model flow. Panel (a) shows full-depth feature selection across three frozen encoders. Panel (b) shows multi-task observations, four latent future states, the planning head, the future-representation objective, and the auxiliary QA path.

Predicting a Future in the Same Coordinates

The future target is constructed from real observations at four delays: 0.4, 0.8, 1.2, and 1.6 seconds after the current frame. For each future time, the model computes the four camera states and applies a projection $G$. With $F,R,B,L$ denoting front, right, back, and left views, the target sequence is:

$$z_k=G\left(\operatorname{Concat}_{v\in(F,R,B,L)}c_{t+\Delta_k}^{v}\right),\qquad k=1,\ldots,4$$

No human label is needed for this target. The target is what the current representation system computes from a real future image. That choice ties prediction to the same feature geometry used by the policy. If the representation changes during training, the future target changes with it. The JEPA loss therefore tests whether the language model can anticipate the evolving state of its own visual interface, not whether it can generate a visually pleasing image.

The language model produces hidden state $h_k$ from the task context and earlier predicted states. A projection maps $h_k$ to $\widehat{z}_k$. The world-model loss averages squared error over all four prediction steps:

$$h_k=\operatorname{LLM}(X,h_{<k}),\qquad \widehat{z}_k=\operatorname{Proj}(h_k)$$ $$\mathcal{L}_{\mathrm{wm}}=\operatorname{MSE}\left(\{\widehat{z}_k\}_{k=1}^{4},\{z_k\}_{k=1}^{4}\right)$$

These latent steps are not written as natural-language chain-of-thought and are not optimized to decode into a fixed vocabulary. They are continuous states rolled out immediately before control. The paper evaluates what they contain by attaching frozen-model probes for future depth, semantics, occupancy, target motion, and ego motion. This separates a measurable internal world state from an appealing visualization.

A predictive target learned online can collapse: if every future state becomes nearly the same vector, the mean squared error is easy to reduce while the state loses information. VISReg constrains the center, scale, and shape of the target distribution. The regularization is decomposed as:

$$\mathcal{L}_{\mathrm{reg}}=\mathcal{L}_{\mathrm{center}}+\mathcal{L}_{\mathrm{scale}}+\mathcal{L}_{\mathrm{shape}}$$

The complete loss combines normalized trajectory error, future-state error, anti-collapse regularization, and answer-token cross-entropy:

$$\mathcal{L}=\lambda_{\mathrm{nav}}\mathcal{L}_{\mathrm{nav}}+\lambda_{\mathrm{wm}}\mathcal{L}_{\mathrm{wm}}+\lambda_{\mathrm{reg}}\mathcal{L}_{\mathrm{reg}}+\lambda_{\mathrm{qa}}\mathcal{L}_{\mathrm{qa}}$$

This objective is not a generic auxiliary loss placed beside a frozen policy. The visual construction layers, language backbone, state token, projectors, and planning head all participate. The QA branch uses the language model's visual input plus features from Depth Anything 3 and DINOv3, but it does not perform the navigation latent rollout. The data mixture therefore tests whether auxiliary language supervision can coexist with navigation without turning the future state into an answer-generation trace.

flowchart TB
    R["RGB history
one or multiple cameras"] --> S["DINOv3
semantic depth stack"]
    R --> G["Depth Anything 3
geometric depth stack"]
    R --> D["OpenD4RT
dynamic depth stack"]
    S --> RR["Three Representation
Residual layers"]
    G --> RR
    D --> RR
    RR --> C["Learnable CLS
one state per view"]
    T["Instruction or goal"] --> Q["Qwen3-VL language model"]
    C --> Q
    Q --> Z["Four future states
0.4, 0.8, 1.2, 1.6 seconds"]
    Z --> P["Shared planning head"]
    P --> W["Eight poses
x, y, yaw"]
    Z --> F["Predicted future targets"]
    F --> J["JEPA error and VISReg"]
    R --> A["QA visual path"]
    A --> Q

Method diagram: The diagram follows the architecture rather than a generic encoder-decoder template. Frozen priors contribute all-depth candidates, the residual layers and CLS token build the self-constructed state, the language model rolls out four future states, and those states feed both planning and the future-target regularizer.

Training Through the Dream

Stage I uses a masked attention curriculum. Visual and language prefixes are isolated, latent positions can read both, and the action position can read the instruction and generated latents but not the full visual prefix. The action loss must therefore pass through the latent future state. Loss weights $(\lambda_{\mathrm{nav}},\lambda_{\mathrm{wm}},\lambda_{\mathrm{reg}},\lambda_{\mathrm{qa}})=(0.01,0.1,0.9,1)$ emphasize representation prediction and anti-collapse behavior.

Stage II restores full causal attention from the Stage I weights. The action position can now use both direct observations and the predicted future. Weights change to $(1,0.001,0.009,1)$, placing the main pressure on trajectory accuracy. The two stages use disjoint halves of the training data, with every task and source represented in both halves. The curriculum tries to establish a useful latent path before allowing the policy to rely freely on all visible evidence.

The language backbone starts from Qwen3-VL-4B. Representation Residuals, the learnable state token, the planner, and both projectors are newly initialized. The three source encoders and the language model's own vision encoder remain frozen. Optimization uses Muon with a $10^{-3}$ learning rate for matrix parameters and AdamW with $10^{-5}$ for the rest, a global batch of 128, and 38,270 steps on 16 GPUs.

The approximately 5.08 million training samples contain 2.52 million vision-and-language navigation samples from R2R and RxR, 0.50 million VLNVerse examples, 0.67 million tracking examples, 1.0 million point-goal examples, and 0.39 million visual-question-answering samples. All navigation data are converted into the same form: an observation history or goal, a task condition, and future waypoint supervision. The unified output does not imply that the tasks are identical; it means they share an action decoder and representation interface.

Benchmark Evidence

The evaluation spans five navigation settings. Vision-and-language navigation is measured on VLN-CE R2R and RxR as well as fine- and coarse-grained VLNVerse. Active tracking uses EVT-Bench. Point-goal navigation uses the InternVLA-N1 protocol and ABotN-PointBench. The metrics include success rate, success weighted by path length, navigation error, normalized dynamic time warping, tracking rate, and collision rate. The diversity matters because a representation that only helps one decoder could still perform well on a single benchmark.

Benchmark and setting	RepresentNav result	Reported comparison	What the result tests
VLNVerse fine-grained, Full Test	83.80 success rate	+20.05 over Qwen-RobotNav-8B	Detailed object and route language
VLNVerse coarse-grained, Full Test	59.00 success rate	+12.41 over Qwen-RobotNav-8B	High-level route language
VLN-CE R2R, Val-Unseen	71.1 success, 66.6 SPL, 3.75 m error	Best R2R success among reported multi-view methods	Continuous action and unseen environments
EVT-Bench distracted tracking, monocular	85.5 tracking rate, 68.3 success	+10.1 tracking points over the best monocular baseline	Moving-target prediction with one camera
InternVLA-N1 point-goal mean	95.0 success, 86.7 SPL	+4.0 points over SPAN-Nav	Home and commercial point goals
ABotN-PointBench, All scenes	97.5 collision-free success, 95.8 SPL	+2.1 on both over ABot-N1 at 95.4 and 93.7	Arrival without collision

Table 1: Selected headline results. Rates are reported on the paper's 0-100 scale. The same action interface supports language, tracking, and point-goal tasks, so the pattern across benchmarks is more informative than any isolated peak number.

The VLNVerse improvement is unusually large for a vision representation change. Fine-grained success rises to 83.80, 20.05 points above an 8B Qwen-RobotNav comparison. Coarse-grained success reaches 59.00, a 12.41-point gain. Fine-grained language requires local object references and exact turns, while coarse language emphasizes landmarks and route topology. The full-depth interface can preserve geometric boundaries and semantic distinctions in parallel, but the paper does not isolate which individual phrase types cause the gain.

On VLN-CE R2R, RepresentNav reports 71.1 success, 66.6 SPL, and 3.75 m navigation error. These continuous-control results are important because waypoint quality, stopping behavior, and accumulated drift matter more than selecting the right room. The model ties Qwen-RobotNav-4B at 71.9 normalized dynamic time warping on RxR, while Qwen-RobotNav-4B remains stronger on other RxR measures. The result is therefore a strong R2R result, not uniform dominance over every language-navigation metric.

EVT-Bench tests active tracking under single-target and distractor conditions. With one camera, RepresentNav reaches 85.5 tracking rate and 68.3 success, exceeding all multi-view baselines on tracking rate. With multiple views it reaches 86.8. TrackVLA and TrackVLA++ still have lower collision rates. This split is technically meaningful: anticipating where a person will move can improve continued tracking without automatically solving safe obstacle avoidance.

Point-goal evaluations expose a different balance. InternVLA-N1 measures mean success across home and commercial scenes, where RepresentNav reaches 95.0 and 86.7 SPL. ABotN-PointBench counts only collision-free arrivals and reports All, Low, and High difficulty columns separately. In the All column, RepresentNav reaches 97.5 success with 95.8 SPL; its Low and High results are 100/97.6 and 95.0/93.9. The near-identical All-column success and SPL values suggest that successful paths are also reasonably direct, but the benchmark protocol still differs from open-world deployment.

Ablating the Space and the Dream

The first ablation removes future prediction and changes only the representation interface. A native vision backbone reaches 79.65 success and 40.80 collision rate. Concatenating all three priors improves this to 87.30 and 28.70. Keeping the same priors but routing them through three Representation Residual layers reaches 93.45 and 18.05. The constructed interface nearly doubles the gain obtained by simply adding the same frozen models.

Configuration	Success rate	SPL	Collision rate	Interpretation
Native vision backbone, no dream	79.65	73.43	40.80	Fixed final visual interface
All three priors concatenated, no dream	87.30	80.20	28.70	More information, fixed composition
Representation Residuals, no dream	93.45	84.28	18.05	Task-shaped multi-depth selection
Full two-stage RepresentNav	95.00	86.70	13.45	Future prediction improves action
Single-stage variant	92.40	83.68	15.15	Open attention weakens latent dependence

Table 2: Component and curriculum ablations. The first three rows hold the future objective fixed while changing only what the policy sees. The last two test whether the action path actually uses the dreamed state during training.

Adding the future objective to the constructed interface raises success from 93.45 to 95.00 and lowers collision rate from 18.05 to 13.45. Removing VISReg produces 92.90 success and 16.40 collision rate, slightly worse than the no-supervision latent. Prediction without an anti-collapse target can therefore create a smooth but uninformative future state. The result also shows why a future-prediction loss should not be judged only by its scalar training value.

The two-stage curriculum reaches 95.00 success and 13.45 collision rate, compared with 92.40 and 15.15 for a single-stage model. To test whether the action still reads the latent sequence, the authors swap latent inputs across scenes or replace them with zero while restoring full visual context. Cross-scene swaps shift the first waypoint by 4.95 cm on average in the two-stage model, versus 1.38 cm in the single-stage model. Zeroing produces shifts of 8.24 cm and 2.68 cm. Normal ground-truth trajectory error is 4.38 cm for the two-stage model; under those interventions it rises to 7.06 cm and 9.76 cm.

A capacity control guards against the simplest explanation. Varying the number of Representation Residual layers keeps success and SPL within about 2.2 points of the default three-layer model. The gain is therefore not explained by adding a few parameters alone. What changes is where the policy can obtain information and how that selection is shaped by its losses.

Reading the Dreamed State

The paper freezes the policy and trains small probes to decode the latent state 0.4 seconds ahead. The probes estimate relative inverse depth, semantic labels, 3D occupancy, and ego motion. None of those quantities is directly supervised through the future target; the target consists of learned state vectors extracted from future RGB. Successful decoding would therefore be evidence that the predictive objective preserved scene structure rather than only memorizing an action code.

Figure 5: Representative future-state readouts. The complete model recovers depth, room semantics, 3D occupancy, and ego trajectory more closely than frozen encoders applied to the real future frame in these selected examples. Table 3 supplies full-test metrics.

Source of future representation	Depth MAE	Occupancy IoU	Semantic mIoU	Ego-motion ADE
OpenD4RT on the real future frame	0.10253	29.79	20.34	0.04713 m
Constructed state from the real future frame	0.07520	35.95	24.82	0.02962 m
Unsupervised latent rollout	0.10680	31.37	18.50	0.05149 m
RepresentNav future latent at +0.4 s	0.09131	34.81	23.38	0.04448 m

Table 3: Probe results on 1,000 test observations per metric. The dreamed latent never receives the future image, yet it carries more usable occupancy and semantic structure than an unsupervised latent and approaches the constructed representation of a real future frame.

Occupancy is the strongest result because no probe target is used during policy training. The full dream reaches 34.81 IoU, above the best pretrained encoder on the real future frame at 29.79 and above the unsupervised latent at 31.37. A plausible state vector must encode where surfaces and free space will be to support waypoint planning, even when no explicit occupancy decoder is part of the training loss.

On 5,268 observations from the expanded Track test set, front-target box IoU rises from 33.69 for the backbone on the real future image to 72.81 for RepresentNav. AP50 rises from 8.67 to 69.44. Future person-motion ADE falls from 1.09701 m to 0.44450 m, and future ego-motion ADE falls from 0.33404 m to 0.15455 m. The latent therefore preserves both target motion and self-motion, two quantities that matter differently for tracking control.

Physical Robots and Runtime

The physical platforms are a Go2 quadruped and a Bumi humanoid. The Go2 carries four SENSING GMSL2 cameras and a reComputer Robotics J3011 with an NVIDIA Jetson Orin Nano 8GB module. The deployed system sends RGB frames to an RTX 4090 server running RepresentNav. The server returns waypoints to the onboard controller, which executes motion. This is a field-connected architecture, not an onboard-only 8GB inference claim.

Figure 14: Hardware and deployment. The Go2 uses four cameras and an onboard Jetson module for sensing and control, while the model runs on an RTX 4090 server. Measured inference is about 144.15 ms for native Qwen, 181.55 ms after adding Representation Residuals, and 209.76 ms for the full model.

The latency increase is moderate at the scale of indoor waypoint control but reveals a real boundary. Representation Residuals add about 37 ms in the plotted comparison, and latent rollout plus planning bring the total to about 210 ms. The number excludes wireless transfer and network variability. A future onboard implementation would need additional compression, a smaller language backbone, or an asynchronous safety controller.

Figure 15: Selected first-person physical episodes across point-goal navigation, instruction following, and tracking. Rows interleave RGB views with semantic, occupancy, target, or latent readouts as the robot moves through an office-like environment.

The quantitative physical evaluation includes 20 language-navigation cases and 20 tracking cases. RepresentNav completes 14 language cases and 18 tracking cases, while Uni-NaVid completes 7 and 8. The reported improvements are 35 and 50 percentage points. These counts are easier to interpret than a highlight reel, although the 20-case sample remains small.

Figure 16: Third-person demonstrations and success counts. The prompts cover following a person, moving around obstacles, entering open areas, turning at corridor endpoints, and passing through doorways. The lower panel compares success counts for 20 language cases and 20 tracking cases.

Limits of the Evidence

The authors state that the method has been validated only on navigation. They note that manipulation, dexterous hands, and humanoid whole-body control remain untested. The architecture is not inherently navigation-specific because the space is shaped by whatever objective the policy optimizes, but that broader claim is a hypothesis rather than an experimental result.

The real-world study contains 40 total test cases, split evenly between language navigation and tracking. The figures show selected episodes rather than every failure. The results establish that the system can run closed loop on physical robots, but they do not estimate robustness across lighting changes, crowded spaces, unusual floor materials, camera contamination, or long-duration drift. The confidence interval around a 14-out-of-20 success rate is wide.

The predictive horizon ends at 1.6 seconds and contains four states. This is reasonable for slow indoor movement, but it cannot represent long-horizon social intent or multi-stage route uncertainty. The paper does not provide a systematic scaling study of the number of latent steps, and it does not measure how prediction errors compound under repeated closed-loop replanning.

Many baseline numbers are taken from cited work rather than rerun under one controlled environment. The paper identifies the source of its comparisons, but hardware, training data, sensor configuration, and simulator details may differ. The internal ablations provide the strongest causal evidence for the representation and curriculum claims; a single benchmark lead is weaker evidence.

The public repository has no implementation or checkpoints at the time of writing. Training scale, optimizer settings, loss weights, data composition, and architecture are described, but the exact future-target construction, masking logic, probe training, and data processing remain unavailable for code-level verification. Reproduction is possible only after those pieces are released.

Conclusion

RepresentNav connects two ideas that are often optimized separately. A compact stack of task-trained residual layers selects useful information across the complete depth of semantic, geometric, and dynamic foundation models. A latent chain of thought then predicts future states in that same learned space, and the shared action head turns the resulting state into waypoints. The future target is not a separate image generator; it is a task-shaped representation built from real future observations.

The experiments show that the interface matters. With the same frozen priors, constructed representations outperform concatenation; with the same constructed interface, future prediction improves success and collision rate; and the latent state decodes into occupancy and motion that were not directly supervised. The physical demonstrations and 210 ms server inference support feasibility. The missing code, modest real-world sample, external compute, and 1.6-second horizon define the next set of questions.

A useful robot world model need not reproduce every pixel of the future; it needs to construct the part of the future on which the next action depends.