L3G-World: Real-Time Infinite World Generation with Latent 3D Gaussian Memory

Background: Gaussians, latent video, and camera control

Each 3D Gaussian carries a center, covariance, color, opacity, and often view-dependent appearance. Its covariance can be parameterized by a scale vector and rotation quaternion:

$$\Sigma=R(q)\operatorname{diag}(s^2)R(q)^T$$

This equation says that an axis-aligned ellipsoid is scaled first and then rotated into world coordinates. Rendering sorts the projected Gaussians by depth and composites them. Compared with a dense voxel grid, the representation can preserve sharp surfaces with fewer primitives and map naturally to fast rasterization.

A video diffusion model instead denoises a latent tensor containing several frames. Text, camera, reference frames, and memory must all guide the same transformer while preserving both appearance and spatial correspondence. L3G-World extends every Gaussian with a 16-dimensional VAE feature. The feature is rendered through the same geometry and alpha compositing as RGB, but it lands directly in the denoiser's latent coordinates.

The generator is initialized from Cosmos-Predict2.5-2B and adapted with LoRA. Camera information enters through UCPE and RayRoPE. The geometry side uses Depth Anything 3 and is inspired by ZipSplat. Training uses four A100 GPUs, with batch sizes of 8 for the memory updater and 16 for the world model.

The method in detail

A memory whose unit is spatial occupancy, not video duration

After the model processes autoregressive window $k$, the persistent memory is written as

$$\mathcal{M}_{k}=\{(\mathrm{G}_{j},\mathrm{T}_{j})\}_{j=1}^{N_{k}}=\{\mathbf{G}_{k},\mathbf{T}_{k}\}$$

Here $\mathrm{G}_{j}$ denotes Gaussian parameters and $\mathrm{T}_{j}$ denotes the corresponding memory token. The recurrence is

$$\mathcal{M}_{k}=\operatorname{Update}(\mathcal{M}_{k-1},\mathcal{W}_{k})$$

where $\mathcal{W}_{k}$ is the current window of images, cameras, and VAE latents. A naive updater could append a new set of Gaussians for every window, making $N_k$ track stream length. L3G-Mem instead decides which parts of the old memory are already sufficient and which parts need new capacity. This is the paper's first main claim: memory should grow with the explored scene, not with the number of chunks alone.

Spawning only where the old world cannot explain the new view

The updater renders the previous memory from the current cameras and compares that rendering with the new observation. Low accumulated opacity means the memory has no reliable surface at a pixel. High RGB error means a surface exists but does not reproduce the observation. The union produces the spawn mask

$$\mathbf{S}_{k}(p)=\mathbb{1}(E_{k}(p)>\tau_E \lor A_{k}(p)<\tau_\alpha)$$

The defaults are $\tau_E=0.05$ and $\tau_\alpha=0.25$. Geometry features inside this mask become spawning candidates, and K-means selects a compact set of query tokens. Clustering all image features would repeatedly allocate primitives to well-observed walls and floors. Restricting clustering to missing or mistaken regions makes the capacity budget follow scene expansion.

Existing Gaussians are allowed to be corrected

A static record would preserve both useful structure and early mistakes. L3G-Mem therefore retrieves previously stored Gaussians visible from the current cameras and sends their tokens through a refinement stream. The visible set is filtered by depth and local opacity. One depth criterion is

$$d_n\leq\hat{D}(u)+\tau_d$$

with $\tau_d=0.05$, and the local opacity threshold is $1/255$. The refinement head predicts residuals for center, scale, quaternion, opacity, color, and VAE feature, with independent learned gates for Gaussian attributes and latent features. Later observations can therefore fix an initially distorted surface rather than merely adding another surface in front of it.

Figure 2: L3G-Mem combines error-aware spawning with retrieved Gaussian tokens for residual refinement. The lower panels visualize how the memory changes as the generated view exposes new evidence.

New and old content exchange information in one transformer

The spawning queries $\mathbf{Q}_{k}$ and refinement tokens $\mathbf{T}^{rf}_{k}$ are concatenated along the token dimension:

$$(\bar{\mathbf{Q}}_{k},\bar{\mathbf{T}}_{k}^{rf})=F_{\theta}([\mathbf{Q}_{k};\mathbf{T}_{k}^{rf}],\mathbf{F}_{k})$$

Inside each transformer block, both streams cross-attend to the current window's geometry features. They then perform stream-wise self-attention followed by global self-attention over all spawn and refine tokens. The global layer is important at boundaries: a newly spawned surface and an existing neighboring surface can exchange information instead of developing independent colors or depths.

Separate heads decode the two token streams:

$$\mathrm{G}_{k}^{sp}=\operatorname{Head}_{sp}(\mathrm{T}_{k}^{sp}),\qquad \Delta\mathrm{G}_{k}^{rf}=\operatorname{Head}_{rf}(\bar{\mathrm{T}}_{k}^{rf})$$

The paper reports 32 Gaussians per geometry token. Residual updates are restricted to retrieved memory tokens; unrelated Gaussians remain untouched. That locality keeps update cost bounded and prevents every new video chunk from rewriting the entire scene.

VAE latents become renderable Gaussian attributes

A 16-channel VAE latent is embedded by a convolutional network into 128-dimensional tokens. These tokens are spatially aligned with geometry and used by a dedicated spawning branch. The spawn head then predicts both geometry and a 16-dimensional latent feature for each Gaussian.

The latent rasterizer shares projection, depth ordering, opacity, and compositing with the RGB rasterizer. However, Gaussian geometry is detached for the VAE objective. The authors deliberately prevent the latent render from moving Gaussian centers: the VAE loss should supervise feature lifting and the shared token representation, while geometry should be learned from the RGB and depth path. The separated gradients are a useful design detail because an unconstrained latent loss could optimize a representation that looks good to the denoiser but no longer corresponds to physical geometry.

Direct latent rendering is not a small implementation preference. In the paper's L3G-Mem ablation, removing VAE lifting drops VAE-space rendering from 943.6 FPS to 2.97 FPS and raises latent MSE from 0.177 to 0.570. The interface choice defines the feasibility of the real-time loop.

Memory conditions the next chunk through channels

At generation step $k$, the previous memory is rendered at the target cameras, producing latent features $\mathbf{z}^{mem}_{k}$ and an alpha map $\mathbf{o}^{mem}_{k}$. The current noisy target is concatenated with these two fields along channels. A fixed reference latent is prepended as a separate token sequence:

$$\mathbf{z}_{k,t}^{seq}=[\mathbf{z}_{k}^{ref};\operatorname{Concat}_{c}(\mathbf{z}_{k,t};\mathbf{z}_{k}^{mem};\mathbf{o}_{k}^{mem})]$$

Reference and target frames have distinct temporal positions:

$$\mathbf{p}_{k}^{seq}=[-L_{\text{ref}},\ldots,-1;0,\ldots,L-1]$$

Only target latents are denoised. Spatial memory is represented by the channel-wise render, not by a new set of transformer tokens for each historical chunk. This keeps the attention sequence from growing linearly with the number of prior windows. The reference latent and temporal RoPE preserve local continuity, while the target-view memory render supplies long-range spatial correspondence.

Figure 3: L3G-Gen conditions a video diffusion transformer on latent features and opacity rendered from memory at the requested cameras. The reference latent remains clean, while only target latents are denoised.

The teacher learns a conditional rectified-flow field

For a clean target latent $\mathbf{z}_{k,0}$ and noise sample $\epsilon$, the interpolated latent is $\mathbf{z}_{k,t}=(1-t)\mathbf{z}_{k,0}+t\epsilon$. The teacher is trained to predict the corresponding flow under memory, reference, camera, and prompt conditions:

$$\mathcal{L}_{FM}=\mathbb{E}\left[\left\|\mathbf{v}_{\phi}(\mathbf{z}_{k,t},t\mid\mathbf{z}_{k}^{mem},\mathbf{o}_{k}^{mem},\mathbf{z}_{k}^{ref},\mathbf{C}_{k},\mathbf{P}_{k})-(\epsilon-\mathbf{z}_{k,0})\right\|_2^2\right]$$

The target learns when local appearance should be copied from the presented world and when the generative prior should invent unobserved detail. Reference tokens are not supervised as future frames, so they serve purely as evidence and temporal context.

Fast inference is distilled on the student's own rollout

L3G-World Fast starts with an AnyFlow flow-map student and then performs on-policy distribution matching against the frozen teacher. The teacher uses 81 frames and 35 diffusion steps. The student processes three latent frames per step in four total steps with no classifier-free guidance.

The student generates its own autoregressive samples, which are re-noised and compared with the teacher. This reduces the mismatch between clean training samples and the imperfect states encountered during fast rollout. Real-time operation therefore comes from three coupled reductions: fewer redundant primitives, direct latent conditioning without VAE re-encoding, and a four-step generator.

The closed loop

Given the initial image, L3G-Mem creates $\mathcal{M}_{0}$. L3G-Gen then renders the current memory from the next target cameras and generates a latent chunk. RGB is decoded once for visualization and geometric supervision. The same generated latent and decoded observations update memory, which immediately changes the condition for the following chunk.

Figure 4: The L3G-World pipeline alternates memory-conditioned generation and memory updating. The persistent representation is both the output of one chunk and the spatial context for the next.

flowchart TB
    A["Single image and target cameras"] --> B["Initialize latent Gaussian memory"]
    B --> C["Render memory VAE features
at next target views"]
    C --> D["Video DiT denoises next chunk"]
    D --> E["Decode RGB once"]
    E --> F["Compare old render
with new observation"]
    E --> G["Retrieve visible old Gaussians"]
    F --> H["Mask uncovered or high-error pixels"]
    H --> I["Spawn compact new tokens"]
    G --> J["Refine retrieved token residuals"]
    I --> K["Joint global self-attention"]
    J --> K
    K --> L["Write updated memory"]
    L --> C

Experiments: what the numbers establish

The main evaluation uses DL3DV and RealEstate10K. Long-horizon consistency is measured on 481-frame loop-closure trajectories: the camera leaves the initial view, explores, and retraces the same path in reverse. Frames at corresponding outgoing and returning poses are compared with PSNR, SSIM, and LPIPS. This protocol targets the behavior that matters for persistent memory, since a model can produce smooth adjacent frames while still inventing a different room when it comes back.

Generation quality uses LPIPS, FID, and four VBench dimensions: imaging quality, aesthetic quality, subject consistency, and background consistency. Camera control uses translation and rotation error. MEt3R measures geometric consistency. Efficiency is split into primitive count, memory footprint, reconstruction throughput, RGB rendering, and VAE-conditioning throughput. The last two are kept separate because a memory that stores only RGB must render and then encode before it can condition the diffusion model.

Long-run GPU cost and throughput

The contrast with NeoVerse is especially large: 7.4 GiB versus 36.8 GiB and 9.02 versus 1.99 FPS over the same 400-frame measurement. WorldKV runs out of memory. These results support the paper's scaling argument, but they are not universal hardware guarantees. The reported FPS depends on resolution, chunk configuration, model variant, and the exact timing scope. The safe conclusion is that the architecture avoids the severe long-run growth observed in the listed baselines under this test setup.

Compact memory and generation quality

Against the most efficient prior explicit 3D memory for each metric, the authors summarize reductions of up to 38.6 times in primitives, a 3.2 times faster reconstruction path, and 359 times faster VAE-space rendering. Those ratios are not three results against one baseline. Each metric chooses the strongest prior comparison relevant to that metric. Against the point-cloud method Lyra 2.0, the paper separately reports 8.9–9.5 times faster reconstruction, 7.8–10.4 times lower memory use, and 643–648 times higher VAE rendering throughput.

The absolute loop PSNR values deserve context. A score of 16.00 on DL3DV and 18.36 on RealEstate10K indicates that returning views are much closer than a drifting model would produce, but it does not mean the generated scene has become a precise multi-view reconstruction. Video diffusion can remain visually plausible while changing fine texture or local geometry. The result should be read as evidence of relative long-horizon self-consistency under the paper's protocol, not as a claim of metric reconstruction accuracy.

Figure 5: Qualitative behavior. Row group (a) shows memory correction after a later chunk reveals geometric error; (b) examines long-horizon consistency; (c) renders VAE memory along diverse camera trajectories.

Component ablations

Disabling adaptive spawning raises the primitive count from 41K to 75K and lowers PSNR from 16.52 to 16.26. More memory is not better when the extra Gaussians repeat already represented space. Removing refinement reduces the quality more sharply to 15.52 PSNR, showing that preserving old primitives without correcting them leaves early mistakes in place. The VAE-lifting ablation has the clearest systems consequence: 2.97 FPS versus 943.6 FPS. The direct latent interface is responsible for the majority of conditioning throughput, not a marginal cleanup.

The world-model ablation tests whether the same memory choices matter after video generation. The full model on DL3DV obtains PSNR 15.11, LPIPS 0.435, and FID 48.09. Removing memory refinement gives 14.82, 0.464, and 50.63. Removing VAE lifting gives 15.41, 0.461, and 51.84. The no-lifting variant's slightly higher PSNR is a good reminder that pixel fidelity and perceptual quality disagree. Its worse LPIPS and FID suggest that RGB-to-latent conditioning introduces a different kind of degradation even when average pixel error remains low. The authors appropriately present the full metric set rather than selecting only the PSNR result.

Figure 6: Loop-closure comparison. Each method starts from the same view, explores unseen space, and returns. L3G-World and its fast variant better preserve structure at return, while several baselines drift, blur, or accumulate rendering artifacts.

A practical assessment for embodied AI

The paper's camera controllability is more relevant to embodied systems than prompt-only video generation. Still, the evaluation supplies one image and a prescribed trajectory. Robot actions, contact, collisions, object state, and physical plausibility are outside the loop. L3G-World answers what the scene looks like along a requested view path much more directly than it answers what happens if a robot pushes, lifts, or opens something. It is best classified as an explorable visual world generator, not yet as a replacement for a physics simulator.

The 481-frame reverse trajectory is a useful stress test because a model cannot hide spatial drift behind local smoothness. Yet this is still a bounded route in mostly static scenes. Frequent occlusion boundaries, fast rotations, highly repetitive texture, pedestrians, and independent moving objects can each invalidate the assumption that a world location has one persistent appearance. Real robot exploration also introduces exposure changes and depth failures that the benchmark does not emphasize.

The efficiency report is transparent in one important respect: it separates reconstruction, RGB rendering, VAE rendering, and final generation rather than combining all work into a vague FPS number. That same separation limits direct interpretation. The H100 400-frame generation result and A100 memory-efficiency measurements use different hardware and timing scopes. A reader should not add the individual throughput numbers to infer a complete embodied interaction rate.

Baseline adaptation also matters. To compare single-image methods under a consistent metric scale, the authors estimate metric depth with Depth Anything 3 and align trajectories; some systems are wrapped. This improves comparability but may move an original method away from its preferred operating condition. Reproduction with the released code and official baseline settings will be necessary before treating the relative ordering as settled.

Limitations

The authors explicitly identify three limits. The memory is designed mainly for static scenes and does not model independently moving objects. Its occupancy still grows under extremely large exploration, motivating adaptive pruning or hierarchical representations. Finally, long autoregressive rollouts can propagate errors into later memory updates; residual refinement can reduce the damage but cannot guarantee correction.

Two additional limits follow from the evaluation. First, a prescribed camera path is a weaker test than closed-loop control. A robot policy may actively choose views that maximize ambiguity or expose previously unseen surfaces, so a next-best-view evaluation would be more demanding. Second, visual self-consistency is not task success. A generated world can return a similar image while violating support, contact, material behavior, or causal relations required for manipulation.

The release state is the immediate practical constraint. Because inference code and checkpoints are unavailable, there is no public way to verify token budgets, K-means query counts, residual gates, memory cleanup, or the claimed timing on a different machine. The project page videos are evidence supplied by the authors, not an independently runnable artifact. Readers should keep that distinction when comparing L3G-World with systems that already expose code.

Conclusion

L3G-World's contribution is systemic. Adaptive spawning decides where capacity is needed; residual refinement prevents old memory from becoming permanently wrong; latent lifting gives the video transformer a spatial interface without an RGB-to-VAE detour. The ablations show that removing any of these changes either compactness, consistency, or throughput in a measurable way.

A natural next step is to separate static geometry from dynamic entities. Objects that move should carry state and pose rather than forcing the static Gaussian field to overwrite itself after every action. Hierarchical memory could also keep high detail near the robot while compressing distant regions. If those layers can preserve identity through interaction and not only through camera return, latent Gaussian memory may become useful for embodied training rather than only for generated exploration videos.

Takeaway

A persistent world model should not remember every generated frame; it should grow memory where space is newly revealed, repair what later evidence disproves, and speak the denoiser's latent language directly.