MemoryVLN: From Ego-Aware Memory Representation to Cross-Context Memory Reuse in Vision-Language Navigation

Paper Metadata

Title: MemoryVLN: From Ego-Aware Memory Representation to Cross-Context Memory Reuse in Vision-Language Navigation

Authors: Xueyao Wei, Yifei Wang, Delun Li, Shaoan Wang, Junzhi Yu, Wenjie Song, Xingyu Chen (Beijing Institute of Technology / Zhongguancun Academy / Peking University)

Links: arXiv arxiv.org/abs/2610.18478 · Code github.com/ZGCA-HMI-Lab/MemVLN · Project page zgca-hmi-lab.github.io/MemVLN · Model card huggingface.co/xiaoxiaoxiaoyyy/MemVLN

Code status: Released, but evaluation-only. The repository ships the closed-loop Habitat VLN-CE evaluation entry point, the memory module, and configuration manifests; training scripts and model weights are not included (the appendix states weights were omitted to satisfy a 50MB upload limit), so reproducing the main table requires retraining from the Appendix A.3 recipe.

One-Sentence Summary

MemoryVLN turns the growing observation stream of long-horizon vision-language navigation into a hierarchical memory: forgetting-curve sampling compresses the sequence, 2D appearance plus frozen 3D geometry plus 6D ego pose stamp every retained memory with where it was seen, an in-episode dual pool and a persistent context-level bank govern storage and reuse, and a Top-K retrieval residually fuses relevant history into the current frame, yielding about 2% and 6.5% success-rate gains over the strongest data-comparable baselines on R2R-CE and RxR-CE.

Background and Motivation

Vision-language navigation asks a robot to reach a goal in an unseen indoor environment using only egocentric RGB observations and a natural-language instruction. The continuous benchmarks R2R-CE and RxR-CE restrict the action space to MoveForward, TurnLeft, TurnRight and Stop, so a single episode routinely lasts over a hundred steps and RxR instructions run to dozens of words. For LVLM-based policies this creates a sharp engineering conflict: the context window is bounded while the observation stream grows without limit.

Existing responses fall into three families. The first feeds every historical frame into the context until memory overflows or attention dilutes. The second compresses history into a fixed-length latent state or a topological map, an irreversible step that can never recover the visual evidence of a critical junction. The third outsources spatial alignment to depth sensors, panoramas or prebuilt semantic maps. MemoryVLN observes that none of these answers three consecutive questions: which observations deserve to survive, how their location is recorded when they do, and how they are fetched back when the current decision needs them.

The paper organizes the answers into two stages. Memory construction covers compression and representation: Long Sequence Compression keeps dense recent evidence and sparse but representative distant observations under a fixed token budget, while Ego-Aware Visuospatial Representation binds ViT 2D semantic tokens, frozen VGGT 3D geometric tokens and a 6D pose encoding so that each memory carries both what was seen and where it was seen. Memory management covers storage and access: a Hierarchical Memory Bank keeps in-episode evidence in short- and long-term pools and environment-level experience in a Context-Level Bank, connected by cross-context warm start and confidence-gated write-back, and Memory Enhancement retrieves the Top-K memories most relevant to the current state and fuses them into the current-frame tokens.

A second motivation is a hard constraint: the method uses no additional navigation data and no DAgger rollouts. Training consumes only standard VLN trajectories, the VGGT geometry branch stays frozen, and the trainable surface is limited to visual projections, memory fusion and the action head. The claimed gains therefore come from how history is organized rather than from data scale, which places the method on a different comparison axis from cross-task pretrained systems.

The boundary with related work is worth stating. World models support planning by predicting futures but do not preserve or retrieve historical evidence. Semantic maps and BEV representations organize landmarks explicitly but usually depend on depth or prebuilt maps and serve episode-specific localization. DUSt3R and VGGT provide strong geometric priors yet remain perception backbones that never decide which observations survive. MemoryVLN's position is to install a frozen geometric prior inside a compressible, retrievable, updatable memory structure so that past observations genuinely participate in the present decision.

Preliminaries

In VLN-CE the agent receives an observation o_t (a single RGB image) and an instruction I at step t and selects a discrete action a_t. Long-horizon decisions require not only the current observation but also evidence accumulated within the trajectory and experience retained from earlier episodes in the same environment. MemoryVLN therefore conditions the policy on two explicit banks instead of the full history.

The backbone is Qwen2.5-VL (7B), mapping instruction and visual tokens to an action distribution; the geometry branch is VGGT, regressing camera poses and point features from multi-view images and kept entirely frozen during training. Evaluation uses NE, OS, SR, SPL and nDTW on the val-unseen splits of Matterport3D scenes rendered in Habitat.

The compression schedule borrows from the Ebbinghaus forgetting curve: retention strength decays exponentially with age, so older observations survive with lower probability. The paper converts this psychological curve into a deterministic two-level sampling budget, allocating frames across chunks by exponential decay and then sampling key frames within each chunk under the same decay.

Method in Detail

Policy: condition on memory, not on history

MemoryVLN defines the navigation policy as

$$a_t \sim p_\theta\left(a_t \mid I,\ o_{\leq t},\ \mathcal{B}^{\mathrm{epi}}_t,\ \mathcal{B}^{\mathrm{ctx}}\right)$$

where o_≤t is the compressed observation sequence, B^epi_t the in-episode bank and B^ctx the context-level bank. Instruction tokens are never compressed; only the growing visual history is selectively retained. Compression, pose injection, memory maintenance, Top-K enhancement and confidence-gated write-back all run online during closed-loop inference. The in-episode bank resets per episode while the context bank persists within an environment scope, an asymmetric lifecycle that stops transient action noise from leaking across episodes while letting reliable environmental experience accumulate.

Figure 1: Conceptual overview. Left, several past trajectories (pink, green) and the current one (blue) share one hierarchical memory bank over the floor plan. Right, the two stages and four modules: construction (compression, representation) and management (bank, enhancement).

Module 1: Long Sequence Compression via two-level Ebbinghaus sampling

The first level fixes how many frames survive. Given N observed frames, the global budget is

$$B(N) = \min\left\{N,\ \max\left[B_{\min},\ a + b\sqrt{N}\right]\right\}$$

so the budget grows sublinearly: about 24 frames at N=100 and 32 at N=200 with a=4, b=2 and B_min=16 (Appendix Table 3). The second level distributes that budget. History is cut into chunks of 8 frames; chunk i has age age_i measured in chunks from the newest one, and its inter-chunk weight and initial allocation are

$$w_i = \exp\left(-\mathrm{age}_i / S_{\mathrm{inter}}\right), \qquad M_i^{(0)} = \min\left\{|C_i|,\ \lfloor B(N)\,\hat{w}_i \rfloor\right\}$$

with ŵ_i the normalized weights and S_inter=3.0 controlling the decay. Leftover budget after flooring is redistributed first by largest fractional part and then round-robin, and no chunk may receive more frames than it contains. The third level samples key frames inside each chunk with probability

$$p_{i,k} = \frac{\exp\left(-\mathrm{age}_{i,k} / S_{\mathrm{intra}}\right)}{\sum_{j \in C_i} \exp\left(-\mathrm{age}_{i,j} / S_{\mathrm{intra}}\right)}$$

where S_intra=5.0 is flatter than the inter-chunk scale so that each chunk still contributes representative older frames. EnsureFirstLast then forces the first and the newest frame to survive: the first anchors the trajectory origin and the newest is the direct evidence for the current decision. Ablation shows that removing compression causes OOM on both benchmarks, so compression is not an efficiency option but a precondition for long-horizon inference at all.

Module 2: Ego-aware visuospatial representation

Retained frames must become memories with a location. The 2D branch supplies ViT visual tokens T^2D, the 3D branch supplies frozen VGGT geometric tokens T^3D, and both are fused with a sinusoidal encoding φ(p) of the 6D pose p=[x, y, z, roll, pitch, yaw]:

$$V = Z^{\mathrm{2D}} + \lambda_{\mathrm{geo}} \cdot \mathrm{Align}\left(Z^{\mathrm{3D}}\right), \quad Z^{\mathrm{2D}}_t = \mathrm{Merge2D}\left(T^{\mathrm{2D}}_t + B(\phi(p_t))\right)$$

The pose enters both streams additively, so two visually identical corridors at different coordinates produce different memory tokens. Restoring pose encoding in the ablation recovers 2.01 SR and 1.01 SPL on R2R-CE with NE down to 3.52, and 1.51 nDTW on RxR-CE, confirming that pose disambiguates observations that look alike but sit in different places. In the released code the encoding is PositionEmbeddingSine6D in position_encoding.py, which raises unless the last tensor dimension equals 6.

Figure 2: Detailed pipeline. (a) Inter-chunk budget allocation plus intra-chunk key-frame selection. (b) Fusion of ViT 2D tokens, VGGT 3D tokens and the 6D pose sinusoidal encoding. (c) Context-level bank with warm start and confidence-gated write-back, plus the in-episode FIFO short pool and cosine-weighted long pool. (d) Top-K retrieval and residual fusion into Qwen2.5-VL-7B for action prediction.

Module 3: Hierarchical memory bank and cross-context reuse

The in-episode bank has two pools. The short pool keeps the 4 most recent frames verbatim in FIFO order; the long pool holds 12 slots where incoming frames are merged into old entries by cosine-weighted averaging with age and usage counters updated accordingly. The cross-episode Context-Level Bank stores up to 300 entries as scene key plus scene value and evicts by LRU when full.

Scene keys mix visual and linguistic evidence so that similar rooms do not collide:

$$k = \mathrm{Norm}\left((1-\beta)\,\mathrm{Norm}(k^v) + \beta\,\mathrm{Norm}(k^I)\right)$$

with the code default β=0.3 (scene_key_lang_weight), i.e. language contributes thirty percent of the key. At the first step of a new episode the bank is queried with a cosine threshold of 0.6; on a hit the retrieved scene value c* warm-starts the initial hidden state by blending with the mean x_0 of the first frame's tokens,

$$h_0 = (1-\tau_w)\,x_0 + \tau_w\, c^*, \qquad \tau_w = 0.5$$

Write-back at episode end is not unconditional. The episode summary enters the scene bank only when the mean frame confidence of the short and long pools crosses δ=0.4:

$$\gamma = \tfrac{1}{2}\left[\mathrm{Mean}(c_s) + \mathrm{Mean}(c_l)\right] \geq \delta$$

The gate keeps badly executed episodes out of the environment's experience. In memory.py, finalize_episode_write_back implements this literally: it computes 0.5*(sc+lc), returns False below memory_writeback_conf_threshold, and can print the skip reason under MEM_DEBUG=1; retrieve_scene_prior implements the 0.6 cosine threshold together with the LRU clock update on a hit.

Module 4: Memory enhancement via Top-K retrieval and residual fusion

Before each decision the frame-level representation q_t of the current frame queries the pools by cosine similarity; the Top-K memories (K=16 by default) receive attention weights and produce an enhanced vector

$$\alpha_{t,j} = \mathrm{softmax}\left(\cos(q_t, m_j)/\tau_m\right), \quad \bar{m}_t = \sum_j \alpha_{t,j}\, m_j, \quad v^{\mathrm{mem}}_t = (1-\rho)\,q_t + \rho\,\bar{m}_t$$

The result is written back residually at token level: tokens are mean-pooled per frame, fusion happens at frame granularity, and the frame-level offset is broadcast to every token of that frame, i.e. Ṽ = V + λ_mem (V^mem − V). This frame-level retrieval with token-level broadcast aligns the retrieval granularity with the memory granularity while preserving intra-frame spatial structure. τ_m=0.2 sharpens the attention and ρ=0.2 (update_ratio) caps how much history may overwrite the present view.

flowchart TD
  O[observation stream o_1 to o_t] --> A[Long Sequence Compression
global budget B N with sqrt growth
exponential inter-chunk quota
intra-chunk key frames, keep first and last]
  A --> B[Ego-aware representation
ViT 2D tokens plus frozen VGGT 3D tokens
plus 6D pose sinusoidal encoding]
  B --> S[In-episode pools
FIFO short pool 4 frames
cosine-weighted long pool 12 frames]
  B --> C[Context-level bank
capacity 300 with LRU eviction
key mixes vision and language]
  C -->|first-step retrieval, cosine threshold 0.6| W[warm start
blend first-frame mean with scene prior
tau 0.5]
  W --> Q[current frame tokens]
  S --> R[Top-K retrieval
softmax cosine over tau_m]
  Q --> R
  R --> F[residual fusion
frame offset broadcast to tokens
V plus lambda_mem times V_mem minus V]
  S -->|end-of-episode confidence gate 0.4| C
  F --> P[Qwen2.5-VL action head
forward left right stop]

The diagram traces the execution order of the method: compression and representation belong to memory construction; the dual pools, the scene bank, warm start, the write-back gate and the Top-K fusion belong to memory management. The two dashed channels, first-step warm start and end-of-episode write-back, are exactly the ingress and egress of cross-context reuse.

Experiments

Main results are reported on the val-unseen splits of R2R-CE and RxR-CE. MemoryVLN uses a single RGB stream, no DAgger and no extra navigation data; baselines follow their published settings, so starred rows that use waypoint predictors or additional data are not fully data-matched. Against JanusVLN, which likewise trains only on VLN-CE expert trajectories, MemoryVLN cuts R2R-CE NE from 5.17 to 3.52 (about 32% lower) and lifts SR from 52.8 to 54.76; on RxR-CE it lifts SR from 51.4 to 57.84, SPL from 44.3 to 50.42 and nDTW from 59.1 to 63.52, which is the roughly 2% and 6.5% success-rate gain quoted in the abstract.

Method	R2R NE↓	R2R OS↑	R2R SR↑	R2R SPL↑	RxR NE↓	RxR SR↑	RxR SPL↑	RxR nDTW↑
Reborn*	5.40	57.0	50.0	46.0	5.98	48.6	42.0	63.3
COSMO	5.58	53.3	47.0	42.7	6.24	48.7	40.9	–
StreamVLN-RGB-only	5.10	64.0	55.7	50.9	6.16	51.8	45.0	62.1
NaVILA	5.22	62.5	54.0	49.0	6.77	49.3	44.0	58.8
InternVLA-N1	4.89	60.6	55.4	52.1	6.41	49.5	41.8	62.6
JanusVLN† (data-comparable)	5.17	58.0	52.8	49.2	6.46	51.4	44.3	59.1
MemoryVLN (ours)	3.52	59.98	54.76	50.26	3.33	57.84	50.42	63.52

Two readings matter. First, the NE improvement dwarfs the SR improvement: 3.52 and 3.33 are at least 1.3 meters below every baseline, so the memory mechanism primarily improves how close the agent gets to the goal and only secondarily whether the stop is judged successful. Second, the absolute SR is not the best in the table: StreamVLN-RGB-only reports 55.7 and InternVLA-N1 55.4 on R2R-CE, both above 54.76, but both consume extra navigation data or cross-task training and therefore sit outside the data-comparable set. The paper's claim is a structural gain under matched data, not an unconditional state of the art.

The ablation removes one component at a time. Dropping Long Sequence Compression causes OOM on both benchmarks and yields no numbers; dropping pose encoding lowers R2R SR to 52.75; dropping the in-episode bank lowers R2R SR to 53.34 and RxR SPL to 48.13; dropping the context bank lowers R2R SR to 54.16. The full model is best on every R2R metric and on RxR SR, SPL and nDTW, and the four contributions are complementary: compression supplies scalability, pose supplies spatial alignment, the in-episode pools preserve trajectory evidence, and the context bank supplies cross-episode reuse.

Variant	R2R NE↓	R2R SR↑	R2R SPL↑	RxR NE↓	RxR SR↑	RxR SPL↑
w/o Long-Seq. Compression	OOM			OOM
w/o Pose Encoding	3.66	52.75	49.25	3.35	57.24	48.74
w/o In-Episode Bank	3.57	53.34	49.06	3.30	56.69	48.13
w/o Context Bank	3.55	54.16	49.65	3.29	57.37	49.09
Full MemoryVLN	3.52	54.76	50.26	3.33	57.84	50.42

Appendix B adds paired per-episode significance tests: on 1,839 matched R2R episodes the SR and SPL gains of 2.0 and 1.2 points are not statistically significant, while on 3,669 matched RxR episodes the 6.2-point SR gain is significant under the exact McNemar test and the 5.2-point SPL gain under the two-sided Wilcoxon signed-rank test. The training-data curve shows the gap widening during training: at 46.4% of the data MemoryVLN reaches 48.6 SR and 44.7 SPL against 44.4 and 37.6 for JanusVLN. Training uses 64 A100 80GB GPUs with global batch 64, bfloat16, AdamW and a cosine schedule with 3% warmup, a single seed 42 and a single reported run.

Figure 3: Four Habitat episodes under identical instructions. Left, JanusVLN deviates from the route, fails to reach the goal or stops incorrectly; right, MemoryVLN enters the target region and stops successfully, with top-down views showing the goal star, the shortest path and the executed trajectory.

Real-robot experiments use a Unitree GO2 quadruped in a real office, with all policy inference on a single RTX 4090. Five instructions ask the robot to stop at a printer, a coffee machine, a designated room, a water dispenser and a conference-room podium; the egocentric sequences show doorways, corridors and office landmarks traversed before a correct stop, indicating that the memory mechanisms learned in simulation transfer to a physical platform.

Figure 4: First-person sequences of the GO2 executing five natural-language instructions in a real office, all ending in a successful stop at the requested target.

Limitations

Author-stated failure modes (Appendix C). The paper selects four reproducible failed rollouts for qualitative analysis. First, room-level semantic branching: instructed to cross the entrance and kitchen into the dining room and stop near the long table, the agent follows the early route correctly but then takes the wrong room-level branch and terminates in a study-like area. Second, fine-grained final-doorway grounding: after the sauna, pool, bar and two stair segments the agent misses the correct first-right entrance, stopping after 60 steps where the reference needs 82, with final NE 10.76 m and Oracle NE 6.08 m. Third, a landmark turn that context memory fails to correct: the agent misses the left turn toward the lamp and drifts into the billiard-table area. Fourth, long-instruction progress loss: on an RxR instruction whose reference rollout takes 204 steps the agent stops after 44 with NE 16.40 m. The authors trace the last case to completed and pending subgoals not being represented explicitly enough, i.e. the memory records where the agent has been but not how much of the instruction remains.

Statistical power and a single run. Appendix A.4 states that all reported numbers come from one training run with seed 42 and no across-run variance, and Appendix B concedes that the R2R gains of 2.0 and 1.2 points are not significant. The headline R2R-CE numbers therefore sit inside noise; the robust gain is concentrated on RxR-CE.

Our assessment: comparison scope and release completeness. First, the "2% / 6.5% over SOTA" phrasing holds only within the data-comparable subset; read as absolute values, R2R-CE SR and OS remain below StreamVLN-RGB-only (55.7 and 64.0), which uses extra data, so the headline must carry that caveat. Second, the repository releases evaluation code and configs only; training code and weights are absent and the HuggingFace card offers no usable checkpoint, so independent reproduction of the main table means re-running a 64-GPU training from the appendix recipe. Third, paper and code disagree on configuration: Table 3 lists 4 short plus 12 long in-episode slots while configure_inference_memory_pool in memory.py uses 8 short plus 24 long at inference, and the B_min=16 floor of Table 3 appears only in the evaluation entry of evaluation.py whereas _compute_global_max_frames inside memory.py floors at 1. Which configuration produced the reported numbers directly determines any reproduction attempt. Fourth, the Context-Level Bank is an in-process list: it is neither persisted nor shared across processes unless DDP sync is enabled, so the lifelong memory the conclusion gestures at does not exist in the current implementation and vanishes when the process exits.

Conclusion and Outlook

MemoryVLN's contribution is not a single module but the decision to treat memory as a first-class citizen in the policy input: compression decides what survives, representation decides how much spatial information survives with it, the banks decide when to store, fetch and evict, and enhancement decides how retrieved evidence enters the current decision. Each module ablates cleanly, maps to a distinct failure mode, and runs online during closed-loop inference without offline mapping or extra sensors. Empirically, the large NE reduction and the statistically significant RxR-CE gains in SR and SPL support the claim that the long-horizon bottleneck is memory organization rather than raw context length, and the GO2 demonstrations give early sim-to-real evidence.

The authors point toward more persistent lifelong memory, more explicit map-like memory and tighter coupling between memory and planning. Judging from the failure cases, the most urgent addition is a memory dimension for instruction progress: the current banks answer where have I been but not what remains to be done, which is precisely why long instructions trigger premature STOP. For the community the paper leaves an open question: once the bank itself becomes part of the policy, should its capacity, eviction and confidence gate be learned end to end instead of being hand-set constants such as 300, 0.6 and 0.4.

Golden Quote

The core bottleneck of long-horizon vision-language navigation is not merely context size but the absence of a hierarchical navigation memory: only once history is compressed, located, managed and retrieved do observations become experience.