GLaM: Training a Latent World Model over Global Spatiotemporal Memory for Active Exploration and Navigation

Paper Metadata

Title: GLaM: Training a Latent World Model over Global Spatiotemporal Memory for Active Exploration and Navigation

Authors: I-Tak Ieong, Ruizhi Feng, Zhaoyang Lu, Yifei Cao, Jiayao Zhao, Leon Li, Senhua Zhu, and Wenbo Ding

Submitted: September 13, 2026　Subject: Robotics (cs.RO)　License: CC BY 4.0

Paper: arXiv:2609.14561　Code status: This reading found no project or repository link in the paper. GitHub searches by title, author names, and architecture terms returned no implementation attributable to the authors. The article therefore cannot map paper components to source functions, and it does not treat an unpublished implementation as verified evidence.

One-Sentence Summary

Core claim: GLaM jointly predicts future map tokens and goal-conditioned robot-centric waypoint latents from an online spatiotemporal map, while the surrounding GLaM Nav system verifies targets with real observations, retrieves remembered locations, and executes geometric motion.

Problem and Motivation

Object-goal navigation asks a robot to find an instance of a requested object category in an unfamiliar indoor scene using only egocentric observations and its own pose. The target may start outside the field of view and may disappear after the robot turns. A practical system therefore has to decide where to search before it sees the target, retain useful evidence from places already visited, and retrieve that evidence after it has left the current view.

Earlier semantic exploration systems, including SemExp, PONI, and VLFM, demonstrate the value of explicit maps, object priors, frontiers, and target-related value signals. Their central question is usually what has already been observed and which frontier or semantic region should be visited next. GLaM changes the learning formulation by asking whether future map representations and future navigation intent can be predicted together from the same persistent memory context.

The word "global" in the title needs a precise interpretation. It does not mean a complete map known before deployment. It refers to the robot's online representation of the space explored so far, which grows as new observations arrive. A prediction can suggest how that representation may extend, but it is not evidence that an unseen room, obstacle, or object actually exists.

GLaM follows a JEPA-like latent prediction strategy. Instead of reconstructing future RGB frames, it regresses future map features and waypoint latents. This shifts supervision away from pixels, lighting, and sensor texture toward spatial representations that are closer to navigation decisions. The tradeoff is that low error in latent space does not by itself guarantee correct target localization, collision-free motion, or a feasible path.

The authors also use cognitive-map concepts as functional inspiration: selective attention, goal maintenance, structured spatial memory, replay, and prospective planning. They explicitly describe this as an analogy rather than a direct model of hippocampal or prefrontal circuitry. That qualification matters because a brain-inspired diagram alone does not establish biological interpretability or a causal explanation of the reported navigation result.

Preliminaries

At time $t$, the predictor receives a history of map tokens, a goal description $g$, and the current robot pose $p_t$. It predicts map-token states and robot-centric waypoint latents over a fixed future horizon. The history length $H$ and prediction length $F$ are fixed by the training setup. This creates aligned samples, but it also means that long-horizon memory must be compressed by the system's mapping and retrieval modules before it reaches the predictor.

A waypoint is not an absolute world coordinate. It is a planar pose increment in the robot's current frame, containing forward and lateral displacement plus heading change. The learned model predicts in a normalized latent space, and a separately pretrained decoder maps the prediction back to a normalized waypoint. A planning-and-control layer then remains responsible for reachability, collision checking, and executable motion commands.

The main simulation metrics are Success Rate (SR) and Success weighted by Path Length (SPL). SR reflects whether the agent stops successfully within the episode budget. SPL also accounts for path efficiency, so a successful but unnecessarily long route receives a lower score. Under the stated ObjectNav protocol, success requires stopping within 1.0 meter of a valid target with that target observable from the stopping pose. The experiment, however, uses a controlled 50% reproduction subset rather than the full benchmark.

Method

Perception, retrieval, and goal verification. The system receives RGB-D observations, pose, and motion state. It encodes visual-semantic features, projects them into the global map representation, retrieves goal-relevant context, and checks whether the requested target has been verified in the current observation or memory. If verification succeeds, the robot can navigate directly to the retrieved location. Otherwise, it enters the predictive exploration branch. This separation prevents a latent forecast from silently becoming an environmental fact.

Structured spatiotemporal memory. The memory combines landmark memory with graph-voxel memory. Selected visual features, depth-projected coordinates, instance information, and poses are associated with spatial units. During an update, the system decides whether a unit should be added, fused, or refreshed. During retrieval, semantic information and associated latent features identify candidate navigation locations. The paper does not disclose the concrete graph schema, matching thresholds, candidate ranking formula, or memory-eviction policy, so those implementation details cannot be reconstructed honestly from the manuscript.

Joint latent prediction. Given the historical map-token sequence, goal, pose, and two sets of learnable queries, GLaM predicts both outputs in one forward pass:

$$ \left(\widehat{\bm Z}_{t+1:t+F},\widehat{\bm E}^{w}_{t+1:t+F}\right) =F_{\Theta}\left(\bm Z_{t-H+1:t},g,p_t,\bm Q^v,\bm Q^w\right). $$

Here, $Q^v$ is the query set for future-map prediction and $Q^w$ is the query set for waypoint-latent prediction. The shared backbone is Qwen3.5-0.8B, and an input projection maps map features into 2560 dimensions. The two heads share the same memory, goal, and pose context. Their joint supervision is intended to place future spatial evolution and intended motion in a common representation space, rather than solving two unrelated auxiliary tasks.

Map-token encoding. Map tokens are built from four RGB-D views. Visual features are projected into a voxel grid and fused within each voxel. Voxel coordinates are expressed in the current robot frame and encoded with three-dimensional spatial rotary positional embeddings (RoPE). The token sequence is truncated or zero-padded to a fixed length, and a validity mask distinguishes real positions from padding. A linear projection then maps the features to the backbone's 2560-dimensional representation.

Waypoint-latent encoding. For a planar increment $(\Delta x,\Delta y,\Delta\theta)$, translation is normalized with training-split scales $s_x$ and $s_y$, while heading is represented by sine and cosine:

$$ \bm u=\left[ \operatorname{clip}\left(\frac{\Delta x}{s_x},-1,1\right), \operatorname{clip}\left(\frac{\Delta y}{s_y},-1,1\right), \sin\Delta\theta,\cos\Delta\theta \right]^{\top}. $$

A two-layer multilayer perceptron encodes this four-dimensional vector into a 2560-dimensional latent:

$$ \bm e^w=\phi(\bm u)\in\mathbb R^{2560}. $$

The encoder has dimensions $4\rightarrow d_a\rightarrow 2560$, followed by LayerNorm. Before world-model training, $\phi$ is jointly pretrained with a lightweight decoder $\rho$ by reconstructing normalized waypoints. Both modules are then frozen. The encoder supplies stable latent targets during world-model training, and the decoder turns predicted latents back into waypoint representations at inference:

$$ \widehat{\bm u}_j=\rho\left(\widehat{\bm e}^{w}_j\right). $$

Freezing this interface makes the supervision target stable and separates prediction quality from decoder quality. It also means that any control-relevant information omitted by the pretrained waypoint encoder cannot be recovered by end-to-end world-model training. The paper does not report the hidden width $d_a$, the amount of encoder pretraining data, or a reconstruction-error acceptance threshold, leaving important reproduction details unspecified.

World-model objective. Training updates the Qwen3.5-0.8B backbone, input projections, learnable queries, and two regression heads. The loss combines future-map regression and waypoint-latent regression:

$$ \mathcal L =\lambda_v\operatorname{MSE}_{m^v}\left(\widehat{\bm Z},\bm Z^*\right) +\lambda_w\operatorname{MSE}_{m^w}\left(\widehat{\bm E}^{w},\phi(\bm U^*)\right). $$

$Z^*$ contains future-map targets produced by a frozen visual encoder and deterministic map-construction pipeline. $U^*$ contains normalized expert waypoints, and $\phi(U^*)$ converts them into target latents. The binary masks $m^v$ and $m^w$ identify valid positions, while $\lambda_v$ and $\lambda_w$ control the two objectives.

For $d$-dimensional target vectors and a binary validity mask, masked mean squared error is defined as:

$$ \operatorname{MSE}_m(\widehat{\bm X},\bm X^*) =\frac{\sum_{\ell}m_{\ell}\left\|\widehat{\bm x}_{\ell}-\bm x_{\ell}^{*}\right\|_2^2} {d\sum_{\ell}m_{\ell}},\qquad \sum_{\ell}m_{\ell}>0. $$

The denominator normalizes by both the number of valid positions and the target dimensionality. The waypoint target uses $d=2560$, while the map target uses the dimensionality of future-map features. Both losses directly supervise continuous representations. The paper states that no language-model cross-entropy term is used. In this architecture, the Qwen backbone is therefore a multimodal latent sequence model, not a text generator that emits navigation answers.

System Data Flow

The following diagram is limited to the paths explicitly described in the paper. It emphasizes that predictions do not write directly into verified memory, and that latent output does not bypass geometric planning and control.

flowchart LR
    A["RGB-D, pose, motion state"] --> B["Four-view features and online voxel map"]
    B --> C["Structured spatiotemporal memory"]
    C --> D["Landmark and graph-voxel retrieval"]
    A --> E{"Target verified from observations?"}
    E -- "Yes" --> F["Navigate to verified location"]
    E -- "No" --> G["GLaM future map tokens"]
    D --> G
    G --> H["GLaM waypoint latents"]
    H --> I["Frozen decoder to normalized waypoints"]
    I --> J["Planning expert and reachability checks"]
    J --> K["Geometric controller"]
    K --> A
    A --> C
    F --> K

The crucial loop is driven by real observations. After execution, new RGB-D data update the map, target verification runs again, and retrieval may change. A prediction supplies prospective context, but it does not update the factual map by itself. This is also why GLaM should be read as a navigation-specific, goal-conditioned predictor rather than a general action-conditioned simulator.

Why the design could help. Future-map supervision asks the backbone to model spatial evolution, while waypoint supervision asks it to represent goal-directed motion. A shared context may encourage both spatial and behavioral information to survive in the same representation. The system also retrieves relevant remembered regions instead of forcing all history through a fixed token window, and observation-based verification reduces the chance that a prediction alone causes an invalid stop. These are plausible mechanisms, not proven causal effects: the current paper does not include component ablations.

Training Data and Protocol

Training trajectories are collected by replaying ObjectNav experts in Habitat with HM3D v0.2 assets. Each episode preserves its scene, initial pose, goal category, and expert rollout. The data record RGB-D observations, robot poses, executed actions, map features derived from observations, frontier progression, and the map updates associated with moving to a frontier. The trajectories are then cut into samples with different history lengths and prediction intervals.

The reported training subset contains 577 ObjectNav episodes across 145 scenes and 55,392 interaction steps. At one observation per second, the authors estimate approximately 15.4 hours of observations. Multi-timescale slicing produces 12,101 prediction samples. HM3D itself has 216 annotated scenes divided into 145 training, 36 validation, and 35 test scenes. The task covers six target categories: chair, sofa, potted plant, bed, toilet, and television.

Table 1. Training resources and evaluation-protocol values reported by the paper

Item	Value	Meaning
Training episodes	577	Replayed ObjectNav expert trajectories
Training scenes	145	HM3D v0.2 training scenes
Recorded steps	55,392	About 15.4 hours at one frame per second
Prediction samples	12,101	Produced by multiple history and prediction windows
HM3D scenes	216	145 train, 36 validation, and 35 test
Evaluation subset	50%	Matched episodes for the reproduced baseline
Success radius	1.0 m	Stop near a valid target with the target visible

The comparison is a controlled reproduction, not a full-leaderboard submission. Both systems run on the same scene distribution, target categories, and task difficulty within the subset. That supports a relative comparison within this setup. It does not support a claim that GLaM beats every published HM3D method, because the paper does not compare directly against methods evaluated on the complete benchmark, another HM3D version, different sensors, or a different protocol.

Simulation Results

Table 2 contains the central quantitative result. GLaM Nav reaches 86.89% SR, compared with 78.50% for the reproduced BSC-Nav baseline. The absolute gain is 8.39 percentage points and the relative gain is 10.69%. On the same evaluation episodes, the integrated system therefore converts noticeably more trials into successful stops.

Table 2. HM3D-ObjectNav results on the controlled 50% reproduction subset

Method	Success Rate	SPL
Reproduced BSC-Nav	78.50%	47.70%
GLaM Nav	86.89%	48.35%
Absolute gain	+8.39 points	+0.65 points
Relative gain	+10.69%	+1.36%

SPL rises only from 47.70% to 48.35%. This is not evidence that GLaM takes much shorter routes. The safer interpretation is that target-search reliability improves substantially while aggregate path efficiency changes only slightly. Without paired trajectories, the intersection of successful episodes, per-category statistics, collision rates, timeouts, and path lengths, the paper cannot show whether the additional successes came through longer exploration or whether GLaM is more efficient on episodes where both systems succeed.

Figure 1. The opening figure presents the functional analogy behind the work: selective attention and goal maintenance, hippocampal-entorhinal spatial memory, predictive simulation, and planning over retrieved working memory. The diagram motivates the architecture; it is not a claim that the network implements biological circuits.

Figure 2 shows the complete loop. RGB-D and pose feed online mapping; retrieved context and map history enter GLaM; the predictor produces future map tokens and latent waypoints; the action expert grounds them through reachability checks and geometric control. Landmark and graph-voxel memories are updated from real observations. This architecture is important for interpreting the result: the reported GLaM Nav score measures an integrated system, not the predictive backbone in isolation.

Figure 2. GLaM Nav's closed loop. The central predictor sees retrieved map context and produces future map tokens plus waypoint latents, while the action expert grounds a subgoal in the observed map and sends commands to the controller.

Real-Robot Demonstration

The physical demonstration uses an Astribot S1 wheeled dual-arm humanoid with an NVIDIA Jetson AGX Orin, an RGB-D camera, LiDAR, and proprioceptive sensing. The authors separate body-independent memory and prediction from the body-specific controller. The predictor communicates through a metric navigation interface, and the platform controller executes the resulting motion.

In the first sequence, the robot searches for a plant in an open-plan office without a pre-built map or target coordinates. Its online memory grows from one node into a connected route as visual landmarks, free space, and poses are associated. In the second sequence, the robot answers that a black trash can is beside the plant, retrieves the previously observed plant location after it has left the field of view, and navigates back to the region.

Figure 3. Real-world deployment. The left sequence shows online exploration and target arrival without a pre-built map. The right sequence shows a spatial-memory query followed by retrieval of the historical plant location and a return route.

The deployment demonstrates that the described interface can operate with real sensing and locomotion, and that remembered object-location relations can support reuse after the target leaves the current view. It is qualitative evidence, however. The paper does not report repeated trials, quantitative success, collisions, timeouts, path length, localization drift, runtime, or tests across multiple buildings, so the demonstration should not be treated as a quantitative transfer study.

Limitations

Author-stated data coverage. GLaM learns from expert-controlled trajectories. Supervision covers states represented in those rollouts, not every state a deployed policy may visit. Recovery from navigation errors, unfamiliar exploration patterns, and alternative routes may require experience absent from the training set. The manuscript discusses the logic of learner-state aggregation but uses a single expert-replay dataset; it does not report an iterative DAgger-style collection round.

Author-stated prediction scope. The model is conditioned on memory, goal, and pose, not an externally specified sequence of future actions. Its output is a goal-conditioned estimate of future maps and waypoints, not a general simulator of arbitrary action-conditioned futures. Calling it a world model is reasonable only with the qualifiers "goal-conditioned," "fixed-horizon," and "navigation-specific."

Author-stated uncertainty and control risk. The model produces deterministic latent predictions over a fixed horizon. Partial observability can admit multiple unseen layouts and feasible routes, but the regression objective does not explicitly represent alternatives or calibrate uncertainty. Low map-feature or waypoint-latent error also does not guarantee target localization, collision-free motion, or consistency between the two predicted outputs. Observation-based verification and geometric control are therefore necessary safeguards.

Author-stated memory fidelity and capacity. Spatial memory depends on visual features, depth projection, and pose alignment. Errors in these components affect both retrieval and prediction context. The persistent map may retain more information than the predictor sees, because the model consumes a bounded token representation and truncation can discard old evidence. How to select tokens in larger spaces, preserve resolution, and control inference cost remains unresolved.

Independent evaluation limitations. The quantitative evidence comes from one 50% HM3D subset and one reproduced baseline. The paper reports no confidence intervals, paired significance tests, category-level breakdown, collision rate, timeout rate, false-stop rate, or runtime. Its ablation section explicitly defers detailed experiments to a later public version. The physical study is qualitative. The manuscript also omits several reproduction-critical hyperparameters, including the frozen waypoint encoder width, loss weights, token limits, and compute budget. These omissions prevent a rigorous assessment of robustness to localization drift, furniture rearrangement, persistent scene change, and substantially larger environments.

Relationship to Prior Work

Compared with explicit semantic exploration systems such as SemExp, PONI, and VLFM, GLaM does not remove the map. It keeps an observation-updated map and trains a predictor over map-level tokens. The emphasis is on future map evolution and waypoint intent rather than frontier selection alone. Compared with ConceptGraphs and BSC-Nav, the paper proposes a tighter connection between structured memory retrieval and a joint latent predictor, instead of using memory only for semantic association or planning.

Compared with latent world models such as PlaNet, MuZero, and DINO-WM, GLaM operates on navigation map tokens and additionally predicts robot-centric waypoint latents. It does not reconstruct future RGB and does not take an explicit future action sequence as input. That narrowing makes the representation more directly relevant to its navigation task, but it also prevents the model from answering general counterfactual questions about alternative actions.

The comparison with BSC-Nav needs particular care. BSC-Nav is reproduced as a strong structured-memory baseline, and GLaM Nav contains its own landmark and graph-voxel memories. Without ablations, the 8.39-point SR difference cannot be attributed specifically to latent future-map prediction. It may come from the predictor, retrieval design, target verification, planning interface, data, or integration effects. The paper appropriately avoids a causal claim, but its architectural narrative is stronger than its current experimental separation.

Implications for Robotics Engineers

First, factual memory and predicted memory should remain separate. GLaM Nav updates persistent memory from real observations and uses observations to verify targets. This is the right default for a physical system: a learned forecast can guide search, but it should not become a map fact without sensing. Second, a frozen latent interface makes supervision stable, but it shifts a large burden to the waypoint encoder. If the encoder omits a control-relevant distinction, later training cannot recover it through the decoder.

Third, success rate alone is insufficient. The small SPL change means developers still need collision counts, time to first verified observation, replanning frequency, timeout rate, false stops, energy, and latency. A system that finds more targets by exploring much longer may be undesirable for many robots. Fourth, fixed-length map tokens are a practical scaling bottleneck. A deployable extension needs hierarchical retrieval or goal-conditioned token selection so that relevant old evidence survives as the map grows.

Finally, a reproduction should start with the missing system contracts rather than only the Qwen backbone and regression heads. Those contracts include four-view voxel fusion, memory matching, candidate retrieval, target visibility checks, action-expert subgoal selection, and the rule for rejecting unsafe latent plans. Without them, an implementation may reproduce the architecture diagram while failing to reproduce the behavior measured in the paper.

Conclusion

GLaM's central technical move is to predict future map representations and robot-centric waypoint latents together from goal-conditioned online memory. GLaM Nav surrounds this predictor with persistent landmark and graph-voxel memory, observation-based target verification, and geometric planning. The training pipeline uses multi-timescale HM3D expert trajectories and a frozen waypoint encoder-decoder interface. On the controlled 50% ObjectNav subset, the integrated system raises SR from 78.50% to 86.89% and SPL from 47.70% to 48.35%.

The evidence supports a reliability improvement on that matched subset. It does not yet establish substantially better path efficiency, a causal benefit from future-map prediction, or robust quantitative transfer to physical robots. The most valuable next steps are therefore not merely a larger backbone. They are released code and hyperparameters, paired trajectory analysis, component ablations, failure-type reporting, repeated real-world trials, latency measurements, and tests under long-term scene change and large-scale memory pressure.

Key Takeaway

"GLaM lets a robot forecast how space may evolve, but the system treats navigation knowledge as real only after observation verifies it."