Papers

LeWAM: End-to-End World and Action Modeling with JEPAs

Joint-embedding predictive architectures (JEPAs) learn compact latent world models without pixel reconstruction. Existing JEPA pipelines typically train visual representations and action-conditioned dynamics before adding a control policy or solver. We introduce LeWAM, an end-to-end JEPA for control with: (1) a joint training objective over next latent state prediction and action flow matching, and (2) an isotropic Gaussian constraint on the latent state, enabling a single model to support direct policy execution and efficient planning by evaluating sampled actions with its learned latent dynamics. Compared with prior JEPAs in visual planning, LeWAM improves success rate from 28.6% to 89.7% on contact-rich tasks and from 10.9% to 46.2% on long-horizon tasks, and plans 32.3 times faster. On behavior cloning tasks in simulation and physical robots, LeWAM is also competitive with established methods. Interpretability analyses show that LeWAM learns control-relevant representations and action distributions effectively guiding test-time planning and control.

JEPAWorld Models

Minghao Fu, Tavis Siebert, Eryk Halicki · 227 views · Oct 8, 2026

Obstacle-Aware Autonomous Coverage and Navigation for Outdoor Robots

Long-duration outdoor coverage with autonomous platforms remains challenging beyond classical planning: deployments face localization drift in open spaces, obstacles in cluttered sites, controller feasibility in turn-heavy maneuvers, and persistent autonomy with energy management. We propose a unified ROS 2 architecture for single-robot outdoor coverage: a dual-antenna RTK-GNSS fused in an EKF keeps both position and heading accurate across long missions; three controller-aware refinements extend a mature coverage planner; a Nav2-based behavior-tree mission executive coordinates multi-goal execution, layered recovery, cost-aware goal management, and autonomous docking for return-to-charge. In real-world trials across five outdoor areas with varying geometries and obstacle densities, the robot completed every coverage route, sweeping 93.1% to 96.1% of the planned coverage area.

Outdoor RobotsCoverage Path Planning

Leonardo Gargani, Matteo Frosi, Matteo Matteucci · 179 views · Sep 1, 2026

Agentic World Action Models

Recent world action models provide visual foresight and future prediction for robot manipulation, but lack explicit mechanisms for compositional skill reuse. Agentic robotics combines code-based planning and reasoning with reusable skills, but often lacks predictive grounding in physical dynamics. We introduce AW-0, an agentic world action model framework that integrates code-based planning with joint video and action prediction. We jointly model code, future video, and robot actions as three coupled generative streams within a unified mixture-of-transformers architecture. Within this architecture a code expert generates parameterized skill routines and control logic while video and action experts generate future visual states and low-level motor commands. Code serves two complementary roles: it structures multi-step prediction and control within the world model and represents skills as reusable programs for subsequent agentic execution. We further introduce an automated curation pipeline that labels robot and human video trajectories with aligned programs and consolidates recurring routines into reusable skill libraries. Evaluations on RoboTwin 2.0 and physical bimanual manipulation tasks show improved task performance over VLA and WAM baselines, alongside gains in language following, semantic reasoning and robustness, demonstrating that agentic code enables connection between planning, prediction and execution within WAMs.

World-action modelsVision-language-action (VLA)

Liu, Zhuoyang, Niu, Dantong, Zhang, Botong · 296 views · Oct 21, 2026

Long Range Navigator (LRN): Extending robot planning horizons beyond metric maps

A robot navigating an outdoor environment with no prior knowledge of the space must rely on its local sensing to perceive its surroundings and plan. This can come in the form of a local metric map or local policy with some fixed horizon. Beyond that, there is a fog of unknown space marked with some fixed cost. A limited planning horizon can often result in myopic decisions leading the robot off course or worse, into very difficult terrain. Ideally, we would like the robot to have full knowledge that can be orders of magnitude larger than a local cost map. In practice, this is intractable due to sparse sensing information and often computationally expensive. In this work, we make a key observation that long-range navigation only necessitates identifying good frontier directions for planning instead of full map knowledge. To this end, we propose the Long Range Navigator (LRN), that learns an intermediate affordance representation mapping high-dimensional camera images to affordable frontiers for planning, and then optimizing for maximum alignment with the desired goal. LRN notably is trained entirely on unlabeled ego-centric videos making it easy to scale and adapt to new platforms. Through extensive off-road experiments on Spot and a Big Vehicle, we find that augmenting existing navigation stacks with LRN reduces human interventions at test-time and leads to faster decision making indicating the relevance of LRN.

Long-Range NavigationOff-Road Navigation

Matt Schmittle, Rohan Baijal, Nathan Hatch · 199 views · Apr 17, 2025

AgenticRoboRL: Bringing Coding-Agent Post-Training to Agentic Robotics

Agentic robot systems pair a language-model planner with a robot harness, but recent gains have largely come from stronger closed-source planners, better harnesses, and external memory rather than from improving the planner itself. AgenticRoboRL investigates whether coding-agent post-training can improve robot planners across both VLA skill invocation and control-code generation. Its recipe combines supervised fine-tuning on successful teacher trajectories with multi-turn reinforcement learning using task success as the terminal reward. A recording proxy preserves each sampled response and its actual context, while harness-specific adapters associate model calls with task outcomes and exclude infrastructure-invalid episodes. Only the planner is trained; perception and execution components remain fixed. With a 9B planner, success rises from 18.1% to 71.0% on LIBERO-PRO and from 19.2% to 72.2% on LIBERO-Plus. With the CaP-X control-code harness, the same post-training pipeline raises success from 2.7% to 28.5%.

Agentic RoboticsRobot Manipulation

Xinxin Liu, Bingxin Xu, Wenrui Bao · 208 views · Oct 20, 2026

From Video to Vision: Learning Generalizable Humanoid Carrying from Human Motion

Most video-driven humanoid imitation policies still need an upper-level module to generate reference trajectories at deployment, and the resulting controllers often depend on global position in a motion-capture studio. MOVER replaces that dependency with a three-stage teacher-assistant-student pipeline. It first recovers human motion from 100 videos with GENMO, retargets it to a 29-DoF Tiangong 3.0 humanoid through GMR, and augments object geometry, scale, and pose to obtain 4,000 reference trajectories. A teacher policy learns strict motion and object tracking; an assistant relaxes global body-position tracking to improve placement generalization; and a DAgger-PPO student removes proprioceptive position and reference-joint-position inputs. At deployment, a head-mounted RGB-D camera and an object tag provide relative object pose, while QuatNet predicts the reference anchor orientation from object pose, joint state, and target location. The paper evaluates four object placements and two shape-size configurations in simulation, then demonstrates carrying on the physical robot without motion capture. Remaining limitations include limited demonstration diversity, imperfect target localization without global proprioception, and a small real-world trial count.

HumanoidObject Carrying

Ziluo Ding · 285 views · Oct 10, 2026

World SLAM Model: Joint World Modeling for SLAM and Navigation

World SLAM Model (WSM) embeds the mechanisms of SLAM, including persistent memory, incremental state updates, and local backend refinement, directly into a navigation world model. A unified Mixture-of-Transformers jointly predicts future RGB observations, dense depth, and camera poses from RGB input, enabling closed-loop navigation with accurate localization and mapping.

World ModelsSLAM

Minghui Qin, Yijun Yuan, Weicheng Zheng · 211 views · Sep 26, 2026

Safe Interval Motion Planning for Quadrotors in Dynamic Environments

Trajectory generation in dynamic environments presents a significant challenge for quadrotors, particularly due to the non-convexity in the spatial-temporal domain. Many existing methods either assume simplified static environments or struggle to produce optimal solutions in real-time. In this work, we propose an efficient safe interval motion planning framework for navigation in dynamic environments. A safe interval refers to a time window during which a specific configuration is safe. Our approach addresses trajectory generation through a two-stage process: a front-end graph search step followed by a back-end gradient-based optimization. We ensure completeness and optimality by constructing a dynamic connected visibility graph and incorporating low-order dynamic bounds within safe intervals and temporal corridors. To avoid local minima, we propose a Uniform Temporal Visibility Deformation (UTVD) for the complete evaluation of spatial-temporal topological equivalence. We represent trajectories with B-Spline curves and apply gradient-based optimization to navigate around static and moving obstacles within spatial-temporal corridors. Through simulation and real-world experiments, we show that our method can achieve a success rate of over 95% in environments with different density levels, exceeding the performance of other approaches, demonstrating its potential for practical deployment in highly dynamic environments.

QuadrotorMotion Planning

Songhao Huang, Yuwei Wu, Yuezhan Tao · 182 views · Sep 16, 2024

Can World and Action Modeling Benefit Each Other under Unified Causal Training?

World and action modeling are closely related, yet their reciprocal benefits under unified causal training remain unclear. We introduce OneWM, a simple framework for unified causal training that treats actions as both conditions for world prediction and targets for policy learning within a shared video-action architecture. Using this framework, we investigate whether learning forward dynamics conditioned on specified actions improves action learning, and whether learning to generate actions in turn improves action-conditioned world modeling. We use LIBERO, LIBERO-Plus, and RoboCasa365 as embodied testbeds to evaluate both policy learning and world modeling, complemented by real-world policy experiments. Learning the visual consequences of specified actions improves policy performance and generalization across changing scenes. In turn, action learning improves multi-view world-prediction fidelity under the same architecture. Together, these results answer both questions positively, supporting reciprocal benefits between world modeling and action generation under unified causal training.

World ModelsRobot Manipulation

Yue Su, Bokai Lin, Xinkai Wang · 231 views · Oct 15, 2026

Make π0.5 Learn to Call Astra for Reliable Robot Manipulation

A frozen pi0.5 policy uses a lightweight CALL head to predict when external reasoning is needed. Astra takes over only for selected recovery events, raising LIBERO success from 95.5% to 97.5% while keeping outside calls sparse.

Vision-language-action (VLA)Robot Manipulation

Jianli Sun, Chengxiang Li, Yonglin Tian · 238 views · Oct 9, 2026

Learning Montezuma's Revenge from a Single Demonstration

We propose a new method for learning from a single demonstration to solve hard exploration tasks like the Atari game Montezuma's Revenge. Instead of imitating human demonstrations, as proposed in other recent works, our approach is to maximize rewards directly. Our agent is trained using off-the-shelf reinforcement learning, but starts every episode by resetting to a state from a demonstration. By starting from such demonstration states, the agent requires much less exploration to learn a game compared to when it starts from the beginning of the game at every episode. We analyze reinforcement learning for tasks with sparse rewards in a simple toy environment, where we show that the run-time of standard RL methods scales exponentially in the number of states between rewards. Our method reduces this to quadratic scaling, opening up many tasks that were previously infeasible. We then apply our method to Montezuma's Revenge, for which we present a trained agent achieving a high-score of 74,500, better than any previously published result.

Reinforcement LearningSparse Reward

Tim Salimans, Richard Chen · 236 views · Dec 8, 2018

Humanoid Mobility via Attention-based Contacts

Humanoid mobility over complex terrain requires accurate foot placement and coordinated whole-body motion capable of robust adaptation. Reinforcement learning (RL) policies trained without motion references generalize well across terrains, yet eliciting effective whole-body coordination through reward design and exploration alone remains challenging. In contrast, reference-based methods yield natural motions but struggle to adapt beyond the predefined trajectories in their training data. We introduce a learning framework that bridges these two approaches by using motion data strictly as a training prior. An explicit, self-supervised, attention-based contact planner predicts future body-environment contacts, while a contact-conditioned tracker translates these plans into coordinated whole-body motion. Together, they establish contact as an interface between sensory observations and control actions. This abstraction allows the policy to internalize the contact-aware structure of locomotion rather than directly replicating kinematics. Consequently, a single policy can exploit data-driven motion priors for natural whole-body coordination while retaining the flexibility to discover environment-dependent behaviors through reinforcement learning. Building on the learned contact planner and tracker, a subsequent fine-tuning stage bootstraps their acquired capabilities to explore environment variations beyond those covered by the original motion data. Operating entirely from onboard observations without any reference inputs at deployment, this framework enables versatile humanoid mobility in challenging environments.

HumanoidLocomotion

Junzhe He, Kehan Wen, Chong Zhang · 429 views · Oct 22, 2026

As an Amazon Associate, we earn from qualifying purchases.