Humanoid robotsEmbodied AIVLM Agent

HomeBody: A Humanoid That Explores, Remembers, and Acts on Its Own

Stanford TML's HomeBody drops the learned VLA middle layer: GPT Astra calls a library of composable skills on a Unitree G1, backed by a digital twin and persistent spatial memory the robot builds by exploring a home it has never seen, then tidies an unseen kitchen and fetches medicine with no task-specific training.

Gio Huh, Cayden Gu, Takara E. Truong, C. Karen Liu, Guy TevetSeptember 30, 202615 min read66 views

Humanoid autonomy has a familiar recipe: a System 2 VLM reads visual observations and instructions, a learned System 1 VLA turns reasoning into commands, and a System 0 controller executes coordinated motion. HomeBody, from Stanford's The Movement Lab (TML), asks a sharper question: as frontier VLMs such as Astra become more capable, is a learned VLA still needed between high-level reasoning and the robot's skills? Can System 2 directly orchestrate a library of reusable motor skills?

Their answer is HomeBody: a system that equips frontier VLMs with persistent spatial memory and composable humanoid skills for long-horizon tasks. In a previously unseen kitchen, a Unitree G1 guided by GPT Astra gathers coffee bags onto the island and discards spoiled milk and orange-juice cartons; it also retrieves a remembered medicine bottle from an underspecified request ("I forgot my medicine, can you get it for me?") and hands it to a person. No environment-specific training data, no additional policy learning.

Architecture comparison: top, the learned action pipeline from System 2 VLM to System 1 VLA to System 0 whole-body tracking; bottom, HomeBody's composable skill interface where a plug-and-play VLM (GPT Astra) sends skill plus target to a skill library (Navigate, Pick, Open drawer) and receives observations and outcomes back — The layer being replaced: a learned action pipeline (top) chains a System 2 VLM to a System 1 VLA and a System 0 whole-body tracking controller; HomeBody (bottom) swaps that chain for a plug-and-play VLM calling a composable skill library, with observations and outcomes flowing back into the next decision (figure: Stanford TML).

01. Two long-horizon demos: tidying the kitchen and retrieving the medicine

Clean up all of the coffee bags and put them in the middle, and throw away all of the milk and orange juice cartons that have gone bad.

Cleaning the kitchen means deciding what to keep, what to throw away and how to move each object to the right place. HomeBody gathers the coffee bags on the island and discards the specified cartons, coordinating repeated trips, grasps and placements across the room. Its view changes with every move, so memory and action feedback help it track what is done and what still needs attention.

Capability tags: multi-object cleanup · task sequencing · navigation + manipulation

Demo one: tidying an unseen kitchen. The G1 shuttles between the refrigerator wall, the sink and the island, concentrating coffee bags and walking spoiled cartons to the bin (video: Stanford TML).

I forgot my medicine, can you get it for me? Also throw out the bad carton while you are at it.

The medicine is initially out of view. HomeBody uses stored keyframes to locate the drawer, retrieves the medicine, hands it to the person, and then discards the carton. Across the room, it uses the right hand to grasp the drawer handle and open the drawer, then the left hand to discard the carton. Selecting between the two arms lets it access targets on both sides of the body.

Capability tags: occluded object · memory retrieval · right-hand drawer opening · left-hand carton disposal

Demo two: retrieving from an underspecified request. The drawer's location comes from keyframes stored during exploration; retrieval, hand-off and disposal chain into one cross-room task (video: Stanford TML).

02. Spatial memory: turning unseen places into addressable locations

Long-horizon loco-manipulation takes a humanoid beyond what its ego view can show. To act across a room, it needs an internal spatial model that connects visible objects with remembered locations and helps resolve ambiguous requests. This context guides where to go, what to act on and how to sequence walking and manipulation. HomeBody combines that spatial memory with composable skills whose execution feedback lets the VLM revise its next decision when an action or transition fails.

Isometric view of the simulated kitchen with three numbered markers pointing at remembered locations (refrigerator wall, sink cabinet, island) and a Now label at the G1's current pose and facing — The context the VLM receives when selecting a skill: numbered markers are locations registered in spatial memory, and Now is the G1's current pose and facing. Memory keeps questions like "where is the drawer?" answerable after the object leaves the ego view (figure: Stanford TML).

Memory takes the form of stored ego keyframes plus the descriptive content attached to each frame, both held in the shared coordinate frame registered by Super Odometry and ICP. The three frames below register the medicine bottle in the drawer, the carton on the counter by the sink, and the person seated at the island waiting to receive the hand-off. At recall time they are presented alongside the current observation, and the VLM decides what happens next.

Ego keyframe: an opened drawer containing a white pill bottle with a blue cap — Keyframe one: the medicine bottle in the drawer. "I forgot my medicine" is executable only because this image, together with its location, was stored (figure: Stanford TML).

Ego keyframe: a red-and-white milk carton standing on the white counter next to the sink — Keyframe two: the spoiled carton by the sink. "While you are at it" needs a remembered object, not a fresh search (figure: Stanford TML).

Ego keyframe: overhead view of a person in a red shirt seated on a stool at the island — Keyframe three: the person at the island. The destination of the hand-off lives in memory too (figure: Stanford TML).

03. Deployment in three steps: explore, Real2Sim, then an everyday task

Step 1 · Explore

You are a kitchen robot, please explore the space!

First, the humanoid is told its role and let loose on an unseen environment. HomeBody collects 0.5× iPhone video, D435i camera observations, LiDAR scans with SLAM, joint poses and waypoints chosen by Astra. Exploration captures the room from the humanoid's own viewpoint, grounding its spatial context in what it can see as it moves and interacts with the space; the context is retained when objects leave the ego view.

A researcher holding a pole-mounted iPhone rig aimed at the Unitree G1 in the kitchen, with the caption bubble We took our Unitree G1 to a kitchen it — Where the 0.5× iPhone video comes from: a phone on a pole follows the G1 into the kitchen. The ultra-wide lens buys observation density across the whole room (figure: Stanford TML).

A real kitchen photograph with four composited Unitree G1 poses along the exploration route past the island, refrigerator wall and bin — Four G1 poses from the exploration route composited into one photograph of the kitchen: Astra picks the waypoints, and the robot turns the room into context with its own footsteps (figure: Stanford TML).

Exploration in progress: Astra's thoughts appear over the ego view as it chooses where to look next, and the robot stores observations while walking (video: Stanford TML).

Step 2 · Real2Sim

HomeBody uses Astra as its Real2Sim agent to build a digital twin in Isaac Sim from the humanoid's own collected data. This grounds its observations in a spatial model of the world, helping the high-level VLM reason about locations beyond the ego view.

Two-stage pipeline figure: stage 1 Explore and reconstruct, with Astra selecting the next waypoint, scene data collection (SLAM plus depth, joint states, iPhone 0.5x video, exploration route) and reconstruction in IsaacLab, MuJoCo and Blender by the Real2Sim agent (Astra); stage 2 Deploy in the real world, with recorded poses and route in the reconstructed scene and a recorded kitchen run by GPT Astra listing Pick, Place, Navigate, Pick, Place with reasons — From exploration to digital twin to deployment: the top half shows reconstruction inputs (Astra-chosen waypoints, SLAM + depth, joint states, iPhone 0.5× video, exploration route); the bottom half shows recorded poses in the reconstructed scene and one recorded kitchen run by GPT Astra (figure: Stanford TML).

Height-colored SLAM point cloud of the kitchen interior outlining the island, stools, cabinets and wall monitors — The height-colored SLAM point cloud: this is the geometry the reconstruction eats, with colors distinguishing points at different heights (figure: Stanford TML).

The reconstruction result in five seconds (video: Stanford TML).

The project page answers "what is required for accurate Real2Sim reconstruction?" with a three-way comparison: human-recorded video alone gives rich appearance but only estimated dimensions; a SLAM reference gives measured geometry but no appearance; HomeBody hands the Real2Sim agent camera observations, measured SLAM geometry, joint poses and selected waypoints, so the twin is geometrically, semantically and visually accurate.

Reconstruction input	Appearance	Geometry	Result
Human-recorded video	Rich visual detail for aligning appearance	Room dimensions estimated from looks	Looks right, measures wrong
SLAM reference	None	Measured geometry constrains dimensions	Measures right, looks wrong
HomeBody (this work)	Robot-collected camera observations	Measured SLAM geometry + joint poses + selected waypoints	Geometrically, semantically and visually accurate

Two matched camera views (refrigerator wall, glass-corner sink) are the evidence: real kitchen photograph on the left, digital-twin render from the same viewpoint on the right.

Real kitchen photograph: white island with sink and faucet, stainless refrigerator and wall monitors behind — Refrigerator wall · real photograph (figure: Stanford TML).

Digital-twin render from the same viewpoint: island, sink, refrigerator and wall monitors aligned with the photograph — Refrigerator wall · digital-twin render from the matched camera (figure: Stanford TML).

Real kitchen photograph: double sink at a glass corner with paper-towel dispensers and a wall monitor reading Course 3: Lettuce with Oyster Sauce — Glass-corner sink · real photograph (figure: Stanford TML).

Digital-twin render from the same viewpoint: corner sink, black dispenser boxes and monitor back panel — Glass-corner sink · digital-twin render from the matched camera (figure: Stanford TML).

Step 3 · Give it an everyday task

Tidy up the kitchen. Put the coffee bags on the island and throw away the spoiled cartons.

An instruction like this is all it takes. HomeBody uses its spatial context to choose actions and targets without an action-level script.

Isometric digital-twin view with several G1 poses distributed along a red route from the island around the sink side and back to the bin — Recorded poses and route in the digital twin during deployment: one cleanup task carries the robot from the island to the sink side and on to the bin (figure: Stanford TML).

In the recorded run shown on the project page, GPT Astra's decision sequence and reasons read:

Pick — "the coffee bag is clearly visible here."
Place — "the held coffee belongs on this counter."
Navigate — "the milk carton was remembered there."
Pick — "the milk carton is clearly visible here."
Place — "the bin opening is visible and reachable."

04. The skill library: five verbs and one shared interface

Navigation, picking, placing and drawer opening form HomeBody's action vocabulary. Skills share an interface for targets and execution results, so the VLM can compose them at deployment; local retries and visual feedback correct execution errors.

Skill	Target specification	Execution notes
Pick	Image point normalized to 0-1000 + which hand	Segmentation, stereo depth, calibrated 3D projection, analytic grasp, spline + IK + swept collision check
Place	Which hand + 3D release target in torso frame + release distance	Moves the held object to the target and opens the hand
Open drawer	Handle alignment target	Handle alignment, hooking posture and backward walking in one skill
Pick from drawer	Image point inside the open drawer	Reaches in and lifts the object clear of the drawer edge
Navigate	2D goal + facing point in map coordinates (meters)	Follows a planned route to a location in the Real2Sim map

The previews below are sped up to about 7-10 seconds; drawer opening and grasping keep their recorded reattempts. The drawer-pick preview starts with the drawer open and keeps the retries continuous, slowing down only for the final successful grasp (14× to 4×).

Pick: grasp and lift the object selected in the ego image (video: Stanford TML, 1.25×).

Place: move a held object to a selected 3D release point (video: Stanford TML, 1.25×).

Open drawer: visually align with the handle, hook it, walk backward; recorded retries included (video: Stanford TML, 1.25×).

Pick from drawer: reach into an open drawer and lift the object clear of its edge, retries kept continuous (video: Stanford TML, 14× to 4×).

Navigate: follow a planned route to a location in the Real2Sim map (video: Stanford TML, 1.25×).

The library is open-ended: connect a learned policy, a classical algorithm or another controller through the shared interface, and HomeBody's VLM can chain it with existing skills to carry out new tasks.

05. From a VLM decision to physical action

HomeBody uses spatial targets to connect task reasoning to physical execution. The VLM selects a skill and its target from the current ego view, map context, gripper state, recalled observations and the previous result, and passes this selection through a structured tool call, leaving the skill to plan and execute the motion. The VLM therefore does not need to know the skill's low-level implementation.

flowchart TD
  A["VLM selects skill + target: image point 0-1000 + hand"] --> B["SAM 2 segments the point"]
  B --> C["Fast-FoundationStereo depth from D435i stereo"]
  C --> D["Camera calibration projects masked geometry to 3D"]
  D --> E["Grasp predicted analytically"]
  E --> F["Minimum-jerk spline reference + IK along path"]
  F --> G["Swept-motion collision clearance check"]
  G --> H["Execute; outcome returned to the VLM"]

Ego view: a milk carton on the counter highlighted with an orange mask and a crosshair on its top face, the point selected by the VLM — The target of one pick call: a normalized image point on the carton's top face, with the segmentation mask following immediately (figure: Stanford TML).

3D view: a red-and-white point cloud shaped like the carton on the counter plane, with a red grasp frame standing above it — What the masked geometry looks like after stereo depth and calibration project it into 3D: the analytic grasp pose stands on top of the point cloud (figure: Stanford TML).

Other skills use targets suited to their actions: navigation takes a 2D goal and facing point in map coordinates, measured in meters; a placing call specifies which hand to use, a 3D release target in the torso frame and a release distance; drawer opening combines handle alignment, a hooking posture and backward walking into one skill, coordinating the transition from reaching to pulling.

Ego view: a row of wooden cabinet fronts with the target drawer panel highlighted orange and a crosshair on its metal handle — The target of Open drawer: a point on the handle. Alignment, hooking and walking backward all happen inside the skill (figure: Stanford TML).

06. Localization and retries: a shared frame and bounded retries

To plan how to complete a task, HomeBody needs to know both where relevant objects are and where the G1 is relative to them. To ground these decisions in a shared coordinate frame, it localizes the G1 with Super Odometry and aligns its SLAM map with the reconstructed simulation using iterative closest point (ICP) registration. Ego camera observations are stored in this shared frame together with descriptive content, so HomeBody can return to recorded positions.

Top-down view of the height-colored SLAM point cloud map: purple floor, teal counters and green wall lines outlining the kitchen — The SLAM map from above, colors separating point heights: one side of the ICP registration (figure: Stanford TML).

Top-down render of the digital twin: beige floor, white island and counters, three round stools — The digital twin from above: the other side of the registration. Only once these two agree do remembered locations and the map under the robot's feet share one coordinate system (figure: Stanford TML).

The target can shift in the camera view as the humanoid approaches. Segmentation identifies the object, and SAM 2.1 tracking with SAMURAI memory selection follows it in subsequent frames; visual servoing uses these tracking updates to correct alignment during the approach, without requiring a new VLM decision for each adjustment.

If a grasp closes without contact, the pick skill can try another grasp candidate or adjust its stance and replan. These local retries are bounded. When recovery is exhausted or the failure needs a different action, the skill returns the reason to the VLM, which can reposition, choose a new target or change its plan.

HomeBody retrying a pick of the pill bottle from a drawer: tracking and visual servoing correct alignment during the approach, and an empty grasp triggers another attempt (video: Stanford TML).

07. Whole-body control and the compute ledger

Long-horizon loco-manipulation requires maintaining balance while the upper body reaches and manipulates. HomeBody uses pretrained AMO for its upper-body-aware lower-body control, which accounts for arm targets when coordinating the lower body. Arm and hand commands run at 250 Hz, with the AMO policy updated every fifth control tick at 50 Hz.

The current skills run alongside perception and motion planning on a single Razer Blade laptop with an RTX 4090 GPU. GPT Astra runs remotely, sending skill requests and targets to the laptop and receiving execution results: local skill execution plus network access to a frontier model makes for a lightweight setup in the wild.

Loop	Rate	Responsibility
Arm / hand commands	250 Hz	Track spline references through grasps and placements
AMO whole-body policy	50 Hz (every 5th tick)	Upper-body-aware lower-body coordination and balance
VLM decisions (remote Astra)	Per skill call	Pick skill and target, read outcomes, replan on failure

08. Scope, costs, and our reading

The project page is candid about scope: Real2Sim reconstruction adds setup time and API costs; task length is constrained by the humanoid's reach, manipulation capabilities and hardware endurance, including finger-servo overheating during extended operation; GPT Astra's reasoning latency introduces pauses between skills; and the current local stack requires an RTX 4090 laptop GPU, with heavier perception models or skills likely needing more compute.

Our reading (added in this edition): what HomeBody really trades is the location of the cost. The VLA route spends on data collection and training and buys cheap inference at deployment; HomeBody spends on a per-room reconstruction and frontier-model API bills, and buys zero task training, since new rooms and new instructions require no further learning. In that trade, spatial memory is the load-bearing wall: without the keyframes, their attached descriptions and the registered coordinates, every VLM decision would stand on the current frame alone, and long-horizon tasks would degrade into a chain of isolated short ones. The strength of the evidence deserves equal candor: the two demos are qualitative success recordings, reconstruction remains a per-room offline procedure, and the pauses between skills are still set by model latency. HomeBody reads as an existence proof that a learned VLA is not strictly necessary, complementary to rather than replacing the route that feeds VLAs with large-scale egocentric data.

flowchart LR
  V["GPT Astra (System 2)"] -->|"skill + target"| S["Skill library (System 0 execution)"]
  S -->|"observations + outcomes"| V
  M["Spatial memory: keyframes + per-frame descriptions + SLAM-registered frame"] --> V
  S -->|"new observations written back"| M

09. Acknowledgments, references and open-source plan

Gio Huh is supported by Caltech as a Mark Reinecke SURF Fellow for research on long-horizon humanoid loco-manipulation at The Movement Lab, Stanford University, and participated in Stanford's Undergraduate Visiting Research Intern (UGVI) program. The authors thank the Stanford Robotics Center (SRC) for providing the kitchen space, TML for API credits, and Alan Yu and Sarthak Kamat for insightful discussion.

References: [1] Super Odometry: IMU-centric LiDAR-Visual-Inertial Estimator for Challenging Environments; [2] NVIDIA Isaac Sim; [3] SAM 2: Segment Anything in Images and Videos; [4] Fast-FoundationStereo: Real-Time Zero-Shot Stereo Matching; [5] SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory; [6] AMO: Adaptive Motion Optimization for Hyper-Dexterous Humanoid Whole-Body Control.

The code is open source under Apache 2.0 at github.com/Stanford-TML/homebody. Per the README release schedule, the MuJoCo SIM backend shipped on 2026-10-05 (reproducing the SRC kitchen long-horizon runs with full physics, Unitree G1 + Dex3 hands, GPU optional), with REAL2SIM and REAL backends planned for 2026-10-12 and 2026-10-19. The README also records a "wrong memory" probe: after the carton was moved, Claude Opus found the moved carton twice, once after 29 minutes.

Alias	Model	Status
astra	GPT-6 Astra	Default, validated
opus	Claude Opus 5.5	Validated
sol / luna / terra	GPT-5.6 Sol / Luna / Terra	Configurable, not validated
fable / sonnet / haiku	Claude Fable 5.1 / Sonnet 5.5 / Haiku 4.5	Configurable, not validated

Source: Stanford The Movement Lab project page, "HomeBody: A Humanoid That Explores, Remembers, and Acts on Its Own" (tml.stanford.edu/homebody/, September 2026), by Gio Huh, Cayden Gu, Takara E. Truong, C. Karen Liu and Guy Tevet; code and README at github.com/Stanford-TML/homebody (Apache 2.0). This article is a full translation and adaptation; all images and videos come from the project page and repository. The Real2Sim comparison table, skill target table, control-rate table, model table and the two flow diagrams are added in this edition.

Humanoid robots Embodied AI VLM Agent Real2Sim Mobile Manipulation Spatial memory Unitree G1 Stanford TML Humanoid

Source:Stanford TMLhttps://tml.stanford.edu/homebody/