ASENA: Self-evolving Agents for Embodied Navigation

Paper Metadata

Title: ASENA: Self-evolving Agents for Embodied Navigation

Affiliations: NVIDIA (GEAR lab), UC San Diego, UT Austin

arXiv: 2609.39207v2  ·  Project page: https://research.nvidia.com/labs/gear/asena

Code status: No public code repository is linked in the paper; the project page above is the only official artifact. The paper does release the full system specification (sensors, safety parameters, instruction format) in its appendix.

Embodiment: Unitree G1 humanoid (simulation: Matterport3D / HM3D)

One-line Summary

ASENA hands a general-purpose coding agent a programmable humanoid robot: the agent reads sensors, writes and repairs programs against recorded execution evidence, and accumulates reusable notes and skills in a persistent workspace, while an optional 4B monocular navigation policy (ASENA-VLN) absorbs low-level motion — all without ever updating a model weight.

Background and Motivation

A useful humanoid must do more than reach a destination. The paper opens with a concrete request: check whether a vending machine contains a specific item and come back with an answer. Fulfilling it requires deciding where to search, recognizing the item, remembering where the user is standing, and communicating the result. Retrieving an object adds grasp planning, pickup, carrying, and handoff. Navigation is only one component of a task whose next step is decided by what the robot observes mid-mission.

A learned navigation policy supplies movement, but it cannot decide what to inspect, how to act on an object, or what to say. Those procedures depend on fresh observations and routinely extend beyond any predefined skill library. This is the gap ASENA targets: procedures that must be constructed and revised during a mission rather than retrieved from a fixed set.

Coding agents are a natural fit because they already write, run, and debug programs against feedback. ASENA's bet is that the same loop — write, run, inspect, repair — can be pointed at a physical robot if the robot exposes sensing, computation, simulation, and action through one programmable interface, and if outcomes are recorded in a form the agent can replay.

Two further requirements follow from physical deployment. First, feedback must be tied to outcomes: when an action diverges from intent, the agent needs recorded evidence to locate and fix the responsible program. Second, useful procedures must persist, so the next mission starts from accumulated experience instead of a blank slate. ASENA answers both with MCAP execution records and a persistent workspace.

Finally, delegating motion to a specialized policy reduces the number of repeated agent decisions and covers behaviors that are awkward to specify geometrically (threading a doorway, aligning to an object). Hence ASENA-VLN, a lightweight 4B monocular policy that remains an optional tool: a program may call it, mix it with geometric navigation, or ignore it entirely.

Preliminaries: The Three Moving Parts

The harness. On the G1, ASENA exposes a 360-degree camera, LiDAR, odometry, a body IMU, and joint states, plus a calibrated head RGB/stereo camera for manipulation. Body motion is executed by the SONIC whole-body controller; optional YAM parallel grippers use a separate controller; speech uses the onboard speaker. Sensing, control, and safety operations are exposed as callable atomic skills.

The workspace. At revision $k$ the workspace is a pair $W_{k}=(N_{k},S_{k})$, where $N_{k}$ holds notes and $S_{k}$ holds executable skills. A fixed coding model consults this workspace while solving tasks. Between revisions a separate fixed-weight improver consumes execution traces $\mathcal{T}_{k}$ and feedback $f_{k}$ to produce the next revision:

$$W_{k+1}=I_{\phi}(W_{k},\mathcal{T}_{k},f_{k}). \tag{1}$$

Model and controller weights stay fixed throughout. In simulation private workers may edit workspaces between passes while writes are disabled during frozen evaluation; on the G1, execution records and optional operator feedback support skill revision while robot motion is disabled.

The policy. ASENA-VLN extends Qwen3-VL-4B-Instruct with dedicated token vocabularies: 880 trajectory tokens and 680 pixel-goal tokens, i.e. $880+680=1343$ added tokens, while visual question answering retains the model's original text vocabulary. It consumes up to eight recency-weighted front-view frames plus a structured JSON instruction and predicts bounded body-frame trajectories.

Method

Grounding programs in robot feedback. The coding agent processes and visualizes raw sensor input to estimate geometry, inspect objects, and monitor execution. Every run is stored as an MCAP record that aligns sensor and robot states with actions, feedback, code, and visualizations. Replaying this evidence is what lets the agent identify where execution diverged from intent and repair the responsible program; the same records feed workspace revision.

Navigation as a choice of tools. The agent can plan motion with geometric tools that build LiDAR-based occupancy maps, propose waypoints, and check obstacle clearance, or it can invoke ASENA-VLN for language-conditioned navigation. The agent decides when to invoke each tool, checks progress through new observations, and revises the plan as needed. This is a deliberate division of labor: geometric tools for structure the agent can verify, the learned policy for behaviors that are hard to specify.

Manipulation. SAM 2 object masks combined with depth measurements ground candidate grasps; the system can supplement geometric proposals with learned candidates from GraspGenX when more options are needed. Inverse kinematics and motion planning turn candidates into trajectories for reaching, lifting, and carrying. Before execution the system rebuilds arm trajectories from measured joint states and checks joint limits, sampled self-collisions, and scene clearance, including held-object clearance during lift and retraction.

Execution checks and safeguards. Generated gestures are validated online in MuJoCo for pose continuity, joint and motion limits, and sampled self-collision; a failed validation returns the violated constraint to the agent for repair. Physical motion requires both validation and operator approval, including motion requested by revised programs. An independent navigation safeguard enforces motion limits and rejects commands when odometry is stale or the battery is low, with heartbeat-triggered stops and manual emergency halts as additional protection. The appendix quantifies these limits precisely: command publication at 10 Hz, translation speeds bounded to 0.12–0.35 m/s, a near-goal speed of 0.25 m/s within 0.5 m, arrival tolerances of 0.10 m and 4 degrees, per-command limits of 2.0 m and 180 degrees, and a 3.0 s active-job heartbeat timeout.

Policy inputs and outputs. The instruction is a structured JSON object carrying a concise navigation command and, optionally, a desired end pose, anchor objects, or motion constraints; a quality field on a 1–5 scale conditions generation, with R2R/RxR evaluation using 5. The shared autoregressive decoder predicts up to eight cumulative body-frame waypoints, each represented as a tuple $(x,y,\theta)$. Fixed bins along each axis convert trajectory tokens into explicit motion targets without a separate continuous action head.

Pixel goals as auxiliary supervision. Pixel-goal tokens encourage the model to localize its intended destination in the image. Crucially, a pixel goal is not directly executable: it requires an additional motion planner or, in simulation, a shortest-path follower. For this reason every evaluation in the paper uses trajectory-token predictions rather than pixel goals. The pixel head is a training signal and a debugging affordance, not the control interface.

Curated atomic-navigation data. A callable policy must follow both extended routes and precise local commands, so the authors curate nine atomic navigation tasks in MP3D and HM3D: moving a specified distance, rotating to a target heading, approaching an object from a given direction, aligning with a target, entering a room, passing through a doorway, following a corridor, and moving between floors. Each task is generated and validated using geodesic distance, visibility, and scene geometry so that every instruction describes a feasible and measurable motion, and each is paired with a waypoint trajectory plus auxiliary pixel-goal supervision.

Training mixture. ASENA-VLN trains on the atomic tasks, short-horizon ObjectNav, and vision-language navigation data from R2R, RxR, ScaleVLN, and SRDF, plus multi-view VQA data to strengthen spatial understanding. GPT-5.5 re-captions the original navigation instructions into the structured JSON format so training matches the callable interface. The reported checkpoint uses fewer than 10K optimizer steps and fewer than 5M training samples including repeats, with a 3:1 mixture of trajectory prediction to spatial VQA. The weighted pools total 28.422M trajectory samples and 9.128M VQA samples.

Deployment on the robot. The policy runs on the G1's Jetson AGX Thor. Quantization matters: PyTorch BF16 sums 2,806 ms of component latency, while FP8 with a reduced vocabulary reaches 877 ms, a $2806/877 \approx 3.20\times$ speedup, with export size dropping from 9.01 to 4.53 GiB. Decoding remains the dominant cost at 683.4 ms, about 78% of the FP8 total. On 256 held-out inputs the FP8-with-BF16-vision variant agrees with the BF16 model on the first waypoint 91.41% of the time and on the exact trajectory 82.81% of the time.

flowchart TD
  A["Coding agent
Codex / Claude Code"] -->|writes program| B["ASENA harness
perception + callable tools"]
  B --> C["Geometric navigation
LiDAR occupancy + clearance"]
  B --> D["ASENA-VLN 4B
optional body-frame trajectories"]
  B --> E["Manipulation
SAM2 masks + depth + GraspGenX + IK"]
  C --> F["Robot execution
SONIC whole-body + YAM grippers"]
  D --> F
  E --> F
  F --> G["MuJoCo validation
limits + self-collision"]
  G -->|approved| H["Physical motion
nav safeguard + heartbeat"]
  H --> I["MCAP execution record
sensors + actions + code + feedback"]
  I -->|inspect + repair| A
  I --> J["Fixed-weight improver
Eq 1 workspace update"]
  J --> K["Persistent workspace
notes N + skills S"]
  K -->|consulted next task| A

The ASENA loop: programs are written against a programmable harness, executed through validated controllers, recorded as MCAP evidence, and distilled into a persistent workspace by a fixed-weight improver.

Experiments

Standalone navigation. Evaluated on R2R and RxR val_unseen with 0.25 m forward steps and 15-degree rotations, ASENA-VLN-4B reaches 3.52 m navigation error, 72.6 OSR, 68.7 SR and 64.2 SPL on R2R, and 3.90 m NE, 73.1 nDTW, 70.2 SR and 59.7 SPL on RxR. On R2R it leads every reported metric. On RxR the picture is more nuanced: it wins clearly on error and route similarity (nDTW 73.1 versus 68.6 for Qwen-RobotNav-4B) but trails that baseline slightly on SR (70.2 versus 71.3) and SPL (59.7 versus 61.5), a tension worth noting against the abstract's blanket state-of-the-art claim.

Standalone monocular VLN-CE results on val_unseen (percentages except NE in metres). ASENA-VLN-4B leads on R2R across the board and on RxR error and nDTW.

Object-goal navigation. Paired with a separate VLM that plans where to search, ASENA reaches 80.8/38.1 SR/SPL on HM3D-v2 and 60.5/33.5, 65.8/37.9, 64.9/38.4 across the seen, synonymous and unseen OVON splits, ahead of every published baseline including ApexNav (76.2/38.0) and ABot-N0 (55.3/32.1 on seen). Consistency across the three category splits indicates the system tolerates variation in both object descriptions and the routes needed to find them.

Agent and policy are complementary. On the 100-task R2R and RxR Agentic Splits, the division of labor is the headline result. With Claude Sonnet 5 as backend, adding ASENA-VLN lifts SR by 11 percentage points on both benchmarks (61.0 to 72.0 on R2R, 50.0 to 61.0 on RxR) and cuts mean tool calls per episode by 59.2% and 62.5%. With the stronger GPT-6 Astra backend the policy adds little success on R2R (87.0 to 88.0) and actually reduces RxR SR from 93.0 to 86.0, yet still halves interaction cost (calls fall 46.9% and 51.0%). Notably, GPT-6 Astra with only geometric and programmatic tools reaches 87.0 SR on R2R, approaching the reported human performance of 90% — evidence that coding agents are themselves strong zero-shot navigation systems.

Agentic Splits (100 tasks each). The learned policy helps the weaker backend's success and every backend's interaction cost.

Workspace evolution without weight updates. Repeatedly presenting the same 100-task subsets with ground-truth feedback, sixteen Claude Sonnet 5 workers execute each pass and a fixed-weight improver updates the canonical workspace between passes. Success climbs from 72% to 98% on R2R and from 65% to 89% on RxR over ten passes while logged worker and improver costs fall. Variants with and without ASENA-VLN converge to similar final R2R success, but the policy keeps a clearer advantage on the longer RxR instructions and maintains higher SPL on both benchmarks — the workspace learns what to do while the policy keeps how cheap.

Figure 4 of the paper: SR, OSR, SPL and logged cost per episode across ten workspace-evolution passes on the R2R and RxR Agentic Splits, comparing runs with and without the VLN tool.

Embodied question answering. Under the Explore-EQA teleportation protocol, where each episode has a nominal step budget of $3\sqrt{A}$ for a scene of area $A$ and normalized exploration steps measure the fraction of that budget consumed, ASENA reaches 81.2% accuracy on HM-EQA and 65.5% on MT-HM3D at high effort, ahead of FAST-EQA (69.2 / 50.5) and Qwen-RobotNav (76.7 / 54.4). At low effort the self-evolved workspace uses only 0.08 normalized steps on MT-HM3D versus 0.19 for Qwen-RobotNav. A paired ablation on 132 MT-HM3D questions shows ASENA-VLN cutting median normalized exploration steps by roughly 19% with little change in accuracy.

Embodied question answering: accuracy and normalized exploration steps. Evolved workspaces trade a little accuracy at low effort for markedly cheaper exploration.

Real-world demonstrations. Four G1 missions run without pre-built maps. In the restroom mission the robot locates doors near the elevator lobby, uses its online map to identify a shorter route, and returns with directions. In the pantry mission it inspects a dispenser and a brewer and reports two coffee machines. In the snack-comparison mission it inspects a reference snack, finds a vending machine, checks whether the same item is available, and returns to answer with a generated whole-body gesture. The fetch-and-handoff task chains navigation, head-camera grasp grounding, an inspected arm-motion plan, YAM pickup, carrying while walking, and an inverse-kinematics handoff above the recipient's hand. All four compose navigation, manipulation and communication through the same programming interface.

Figure 5 of the paper: four G1 demonstrations pairing online maps and recorded trajectories with observations and actions; blue and dashed orange traces mark search and return legs.

Figure 3 of the paper: the shared Qwen3-VL-4B decoder with separate trajectory and pixel-goal token blocks, alongside recorded atomic training examples with overlaid waypoints.

Limitations

Author-stated. The G1 demonstrations provide qualitative evidence of capability composition rather than a systematic measure of reliability across objects, layouts, and users. Real-world tasks take 10–20 minutes, making faster reasoning and execution important for practical use. The authors also flag that retained skills have not yet been evaluated on new tasks, that navigation and manipulation reliability over repeated real-world trials remains unmeasured, and that safeguards for dynamic environments need strengthening.

Assessment. Two further caveats stand out from the numbers themselves. First, the workspace-evolution gains are measured on recurring subsets: the same tasks, scenes, instructions and initial poses are revisited across passes with ground-truth feedback, so the 72-to-98 and 65-to-89 improvements demonstrate retention on repeated tasks, not yet transfer to unseen ones. Second, the policy is not uniformly beneficial: on RxR with the strong GPT-6 Astra backend, enabling ASENA-VLN lowers SR from 93.0 to 86.0, and in the standalone RxR comparison it trails Qwen-RobotNav-4B on SR and SPL. The value of learned navigation is backend-dependent, and the paper's own tables make that visible.

Summary and Outlook

ASENA's contribution is architectural as much as algorithmic: it treats a humanoid robot as a programming environment with recorded, replayable outcomes, and treats improvement as revision of notes and skills rather than of weights. Within that frame, a 4B monocular policy trained on geometry-derived atomic tasks becomes a composable tool that raises a weak coding backend's success by 11 points and halves every backend's interaction cost, while workspace evolution lifts recurring-task success by 26 and 24 points on R2R and RxR with all weights frozen. The stated next steps — evaluating retained skills on new tasks, measuring real-world reliability over repeated trials, hardening safeguards for dynamic scenes, and extending evolution to the policy and agent themselves — are precisely the experiments that would turn a compelling systems result into a general recipe.

Golden Quote

"Navigation is therefore only one part of a larger task whose next step depends on what the robot observes."