Papers

Social-WM: Safety-Aware Latent World Models for Robot Social Navigation

Safe social navigation requires a robot to anticipate not only the future consequences of its actions, but also whether a nominal action can actually be executed under surrounding physical and social constraints. We present Social-WM, an efficient latent world-model planning framework trained from egocentric RGB video sequences. Our key observation is that social-navigation experience contains a systematic discrepancy between the nominal action and the realizable action: a nominal forward action may be fully executed in free space, but needs to be constrained when heading towards a pedestrian or obstacle. Social-WM learns these safety-relevant consequences directly through action-conditioned future prediction, where the target is the actual observed future following each command. We further introduce a realizable inverse-dynamics objective that associates observed latent transitions with the action actually realized rather than the nominal one. At deployment, candidate actions are imagined through the latent world model, and the inverse dynamics model estimates their realizability; nominal–realizable discrepancy then provides a safety signal before execution. The learned dynamics and realizability model remain goal-independent and support both position- and image-goal navigation. On Social-HM3D, Social-WM achieves 63.77% success while reducing human collisions to 21.67%, and maintains strong performance under zero-shot transfer to Social-MP3D, without explicit pedestrian tracking, privileged human state, or online reinforcement learning.

Social NavigationWorld Models

Zhihao Zheng, Mooi Choo Chuah · 279 views · Sep 30, 2026

Juno: Taming Predictive Latents for Vision-Language-Action Models

Joint-embedding predictive architectures (JEPAs) predict masked or future observations in representation space, offering a natural source of predictive latents for vision-language-action (VLA) models. Yet making these latents useful across pretraining, policy learning, and deployment requires addressing three failures: mismatch with embodiment-specific control, interference with action learning, and teacher miscalibration under distribution shifts. We introduce Juno, a unified framework built around one action-conditioned JEPA that serves as a control-aligned representation backbone, a predictive teacher, and an adaptable dynamics model. During pretraining, we train it on embodiment-matched trajectories and use a dynamic CLS loss to transfer motion-weighted patch dynamics to a compact global state. During policy learning, we fuse current-frame JEPA patches into VLA perception and use a decoupled reasoning branch with separate transformation parameters to distill future latent states for action generation. During deployment, we adapt the world model on all observed transitions, including failed rollouts, freeze the adapted teacher, and re-align the policy on verified executions using LoRA adapters and a trainable action head, without expert corrections or task rewards. On SimplerEnv, Juno raises average success from 60.9% to 68.5% over Qwen3GR00T, the strongest baseline, and test-time adaptation further reaches 72.7%; on a real robot, it retains 70%–75% success under background, height, and object shifts where the base policy collapses to 0%.

Vision-language-action (VLA)World Models

Yuchen Zhu, Chenyi Xu, Yulin Zhang · 248 views · Oct 7, 2026

Into the LidarVerse: A Real-Time Unified LiDAR World Model Across Tasks and Sensors

We present LidarVerse, a real-time unified LiDAR world model for sensor simulation across tasks and sensors. Existing LiDAR simulators largely follow one of two complementary paradigms: reconstruction-based methods preserve the geometry of a captured scene but require lengthy per-scene optimization, whereas generative methods amortize learning across data but lack the consistency needed for long-horizon rollouts. LidarVerse brings both paradigms into a single, unified conditional generative framework. A universal tokenizer maps LiDAR range sequences into a shared latent space, where a bidirectional teacher learns a common interface for conditioning on a LiDAR prefix, an HD-Map, a persistent spatial cache, and optionally, text. Varying the available scene evidence turns the same model into a generator of new worlds, a point-cloud forecaster, or a generalizable resimulator of observed scenes without per-scene optimization. We distill the teacher into a four-step autoregressive causal student and maintain both a rolling temporal K/V cache and a world-aligned spatial cache, enabling persistent long-horizon rollouts with real-time inference at 11 FPS on a single H100 GPU. LidarVerse achieves state-of-the-art performance on most reported metrics across all three evaluation protocols, while its tokenizer supports 32-, 64-, and 128-beam LiDAR and generalizes to an unseen scanning pattern. These results demonstrate that generative and reconstruction-grounded LiDAR simulation can be realized within one model, providing the core capabilities for future closed-loop sensor simulation.

LiDARWorld Models

Che, Henry, Ren, Xuanchi, Fidler, Sanja · 385 views · Oct 13, 2026

RSI-Anything: Unlocking Recursive Self-Improvement with World Models

RSI-Anything turns predictive world models into the steering signal of a recursive self-improvement loop built on JEPA-Anything. Predictions decide which candidate actions earn scarce external feedback; that feedback then drives policy learning or workflow revision and, when reliability checks show the predictor has drifted under the changed policy, conditional world-model adaptation, while component and compatibility gates decide which policy-model pair is retained to shape the next round. With the coder held fixed, hidden-test success on 200 CodeContests tasks rises from 26.3% under a capacity-matched static verifier to 29.2% with a fixed world model and 31.2% after adaptation, at comparable online test usage. GUI evaluation reaches 99% on MiniWoB++ click-checkboxes while cutting mean actions per episode by 39.4%; five successive laboratory-planning policy versions more than double the LabHorizon F200 Level-2 score from 21.80% to 46.01%; and in a 40-experiment Suzuki-Miyaura flow screen RSI-Anything reaches 96.15% mean best observed yield against 82.16% for Bayesian optimization. An end-to-end immune-science application carries the same retained-workflow-plus-predictor contract into prospective therapeutic hypothesis generation.

Recursive self-improvementWorld Models

Taoyong Cui, Pinxian Zeng, Wanghan Xu · 539 views · Oct 9, 2026

AgentGarten: Code Worlds for Evolving Agents

Interactive virtual worlds allow agents to learn through exploration and interaction. What agents can learn is bounded by the environments they practice in, which must be faithful, with consistent state, rules, and dynamics, and realistic, with observations that follow the real-world visual distributions. Achieving both across diverse worlds remains a bottleneck. We introduce AgentGarten, a framework that couples simulators and game engines with a shared neural renderer to build real-time interactive environments. Its simulation backends maintain persistent world state and execute program-defined interaction rules, while the renderer generates visual observations from structured conditions exported through a common interface. To build the neural renderer, we adapt a pretrained video model to geometry conditions, distill it with our proposed Adversarial Forcing, and optimize inference for real-time interaction. Adversarial Forcing makes history prefilling differentiable through exact replay, so that losses on later predictions update how the renderer encodes prior observations, and adds real-data adversarial supervision to improve its visual quality. In AgentGarten, agents perceive the world through visual observations, interact with it in real time, and improve by distilling each round of experience into playbooks that subsequent agents inherit and refine. Our empirical study demonstrates a substantial gain in learning efficiency, with agents learning from just 4 rounds compared with millions for a conventional reinforcement learning counterpart. As new worlds can be written as code and rendered through the same interface, environments can scale in both number and difficulty alongside their agents, a step toward agents that keep evolving through interactive experience.

World ModelsNeural rendering

Jiawei Chi, Shangchen Miao, Zhiyuan Shi · 675 views · Oct 8, 2026

RIWANav: Recursive World-Action Models with Self-Improvement for Urban Navigation

Long-horizon urban navigation compounds single-step errors, imitation learning only supervises expert successes, and a frozen action-conditioned world model drifts out of alignment as the policy evolves. RIWANav casts the coupled adaptation of a world model and a navigation policy as task-specific recursive self-improvement: Cosmos imagines 17 future frames for four sampled waypoint chunks per state, the resulting scores drive GRPO with a KL penalty and an expert anchor, and the improved policy then proposes actions whose expert-consistent action-video pairs (weighted ADE/MADE/AOE/MAOE consistency Q >= 0.35) enter a policy-induced replay pool prioritized by behavioral novelty and current prediction error to refine the world model over K=19 rounds. On UrbanNav it reaches 89.92% SR / 86.87% SPL on test-seen and 88.25% / 82.59% on test-unseen, beating frozen-world-model GRPO-WM by 7.94 / 6.50 points and cutting navigation error from 3.3314 to 2.0429, with world-model L1/LPIPS on policy-induced actions down 3.2% / 7.1%. Real-world trials on a custom mobile robot confirm practical transfer.

UrbanNavigationWorldModel

Jing Xie, Shouwei Ruan, Yubin Wang · 173 views · Oct 6, 2026

WorldKV: managing a world model's memory like a cache

Memory in autoregressive video world models is a bind: full KV attention preserves revisit consistency, but memory footprint and attention cost grow linearly with rollout length, passing 200 GB per minute and exceeding a B200. Sliding window is real-time but drifts. WorldKV trains nothing and works purely at inference: it retrieves evicted KV chunks by camera/action similarity and compresses each chunk to half size by key-key cosine similarity, fitting roughly twice the history at roughly twice the throughput of full KV, with no memory training at all.

WorldModelKVCache

Jung Yi, Minjae Kim, Paul Hyunbin Cho · 514 views · May 21, 2026

SparseVideoNav: Real-Time Navigation Video Generation with Sparse Historical Conditioning

SparseVideoNav from OpenDriveLab, HKU and collaborators replaces dense full-frame conditioning with sparse history frames (1, 2, 5, 8) for navigation video generation: latent concatenation plus a zero-initialized history cross-attention branch preserves long-horizon memory, and a three-stage curriculum (full-frame pretraining, sparse fine-tuning, PCM distillation) compresses sampling to 4 steps with 0.4 s per-frame latency, 21% lower FVD than the strongest baseline, and behavior-cloning success lifted from 54% to 67% by generated variants.

World ModelsNavigation

Hai Zhang, Siqi Liang, Li Chen · 556 views · Feb 5, 2026

SparkDiffusion: Mitigating the High-Sparsity Trap — A Unified Framework for up to 265× Single-GPU Acceleration of Visual Generation

SparkDiffusion (arXiv:2609.23153) from Peking University, Tsinghua and Alibaba diagnoses the high-sparsity trap: at 97% attention sparsity, step-local training loss keeps falling while terminal video quality stagnates or degrades — the root cause is supervision, since flow-matching-style step-local losses cannot constrain terminal errors (an oracle probe shows fixing the five highest-noise steps removes most terminal error). The three-stage recipe: a short sparse warm-up (compensated sparse attention, RoLA), few-step trajectory-mixed distillation (CrossDistill: high-noise PCM consistency + low-noise DMD distribution matching, 3-step CFG-free student), and FP8 W8A8 quantization with fused kernels. It sustains 97% sparsity on long 720P generation, achieving a measured 265x end-to-end speedup on Wan2.1-T2V-14B-720P on a single RTX 5090 (220x on H100) and 1.3 s for 1.3B-480P videos, with seed-level diversity closest to the dense reference (VBench 83.15 vs 83.69). Code open-sourced.

Video GenerationDiffusion models

Yuxi Liu, Haoyu Li, Zekun Zhang · 732 views · Sep 19, 2026

CAP: Continuously Adaptive Perception-Blind Humanoid Locomotion via Learned Denoising

Humanoid locomotion across complex terrain demands forward-looking exteroception to anticipate obstacles, yet this signal is unreliable in real-world deployment, failing partially and intermittently. Existing perceptive policies often assume that depth observations remain clean and in-distribution, while recent attempts to unify perceptive and blind control typically route or switch between separate sub-policies, leaving recoverable information in partially corrupted depth unexploited. We instead propose CAP, a single-stage humanoid locomotion policy that recovers this signal with a perceptive world-model encoder trained as a learned denoiser to reconstruct clean depth from a corrupted input, together with a co-active proprioceptive variational encoder that supplies depth-free body-state information. A coupled training recipe pairs a depth-noise curriculum on the world-model input with world-model feature dropout on the policy-facing latent, exposing the policy to failures across the entire perception-quality spectrum. In simulation, CAP matches or improves upon perceptive baselines when depth remains informative, and degrades more smoothly than a binary-switching baseline as perception worsens. On the Unitree G1, controlled trials and indoor-outdoor deployments demonstrate perception-robust locomotion under intermittent occlusion, real-sensor corruption, and outdoor depth artifacts.

Humanoid LocomotionWorld Models

Chen, Hongjin, Xu, Zijun, Ma, Shihao · 1.4k views · Sep 10, 2026

JEPA-Anything: Learning Predictive Models across Different Worlds

JEPA-Anything introduces orthogonal predictive factorization for domain-agnostic world modeling, splitting latent targets into complementary factors with dedicated predictive pathways. Across seven domains, it improves matched dynamics tasks and supports intervention, OOD generalization, and long-horizon forecasting.

World ModelsJEPA

Cui, Taoyong, Wang, Zhongyao, Xu, Xinyue · 1.1k views · Sep 17, 2026

Dream-RSI: Recursive Self-Improvement through Evolving Worlds

Recursive self-improvement is becoming increasingly vital for autonomous AI agents, where progress hinges on discovering high-value solutions across complex domains. The driver of this process is effective exploration, however, managing and improving exploration strategies remains a major bottleneck. Current systems face a fundamental dilemma: fixed strategies fail to adapt as search spaces scale, while online policy optimization requires navigating vast meta-search spaces under delayed and expensive feedback over long-horizon rollouts. We introduce {Dream-RSI}, a framework for scalable and recursively self-improving exploration. A lightweight orchestration layer makes exploration explicit and programmable while leaving the underlying coding agent unchanged. Our key insight is that accumulated discovery history can serve as a replay simulator over the realized search space. By performing dreaming in the replay simulator constructed from historical discovery trees, {Dream-RSI} secures immediate, low-cost off-policy feedback to evaluate and refine exploration policies without invoking repetitive, expensive online evaluations. The improved policy is subsequently redeployed online to drive further discovery, continuously expanding the simulator pool in a self-improving loop. Across algorithm engineering, mathematical optimization, and GPU kernel engineering, {Dream-RSI} achieves competitive or improved discovery quality while substantially reducing discovery cost in several settings.

Recursive self-improvementWorld Models

Tong Zheng, Xidong Wu, Zheng Zhang · 1.2k views · Sep 14, 2026

As an Amazon Associate, we earn from qualifying purchases.