InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants

Paper Metadata

Title: InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants

arXiv: 2610.33966 · Code: released at github.com/facebookresearch/InteractiveWorld · Project page: meta-interactiveworld.github.io · Benchmark: InteractiveBench, 120 tasks across 6 categories and 9 capability checks

Environment: Isaac Sim with over 8,000 objects and 50 scenes (44 of 51 layouts pass task admission) · Protocol: twelve frontier models serve as the Assistant, Gemini 3.1 Pro as the Human in the main experiments

One-Sentence Summary

InteractiveWorld closes the loop between an MLLM assistant and an MLLM-played human inside Isaac Sim: the assistant sees only first-person frames and the conversation, the human turns advice into 36 grammar-exact action primitives, and success is read off real object states in the simulator, where even the strongest of twelve frontier models completes only 41.7% of tasks.

Background and Motivation

Most evaluations of multimodal language models are static. A video, a set of egocentric screenshots, or a textualized history goes in; an answer comes out. Nothing the model says changes the world, and nothing tells the model that its previous suggestion was wrong and the scene has moved on. Such benchmarks measure perception and recall, not assistance.

Real assistance is a loop. When you help a friend find something in a kitchen, every sentence you utter changes where he looks and what he touches next; when he turns around or opens a cabinet, your visual information updates, and you must remember what you said and what he did before the next sentence can be useful. Advice, execution, feedback, revised advice: break any link and the assistance fails.

No existing embodied environment covers this chain. BEHAVIOR-1K-style environments target robots or task executability and abstract the human into a directly controlled executor; egocentric memory benchmarks offer first-person views without an executable action interface; dialogue benchmarks offer turns without a world. A scripted human is worse than useless here: if the human follows a screenplay, the assistant's advice never actually affects anything, and the loop is a fiction.

Making the loop real poses three engineering challenges the authors name explicitly. Physical stability: a physically simulated human falls over and wedges into walls at random, so the same advice executes differently twice and nothing is reproducible. Behavioral naturalness: the human must navigate and reach on its own, because teleporting removes the cost of the words walk over. Semantic executability: natural-language advice must land on a deterministic command grammar, and the outcome must be verifiable from environment state.

InteractiveWorld answers with a three-layer architecture, a two-MLLM closed-loop protocol, and a fully state-scored benchmark called InteractiveBench. The question it quantifies is blunt: once the loop is real, how far are today's strongest models from reliably helping a person finish a task? The answer is: far, and the failures concentrate in a few places.

Preliminaries

The SMPL+H body. The camera wearer is a fixed-shape SMPL+H model: a 52-joint kinematic tree with 6,890 vertices and 13,776 triangles. Rendering uses the full surface mesh while collision queries use a compact proxy rig of 17 capsules (pelvis, spine, neck and collars, upper and lower arms and legs, plus hand capsules extending from the wrist along the fingers). The split is one source of stability: the body must look human visually but must never self-intersect physically.

Kinematic, not dynamic. The body accepts pose sequences, not forces or torques, and the physical layer guarantees that any valid pose sequence executes deterministically. The upper layers therefore never handle a branch called the human fell down; evaluation variance comes from model decisions, not from the solver.

Two calibration endpoints. The same Human agent paired with an Oracle Assistant that emits the correct command sequence passes 100% of tasks; with no assistance at all it passes 0%. These endpoints fix the scale of the benchmark: every task is completable by the human, every task genuinely requires help, and model scores fall between the two with a clear meaning.

Method

Three layers. Bottom up: the physical layer turns pose sequences into stable, repeatable human motion and first-person frames; the human action layer turns goals such as walk to the fridge or grab the cup into navigation paths, stances, grasps and full-body poses; the semantic layer translates between natural-language advice and a deterministic command grammar and maintains the turn protocol. Only with all three is the human an interactive agent that observes, understands and executes, rather than a remotely controlled character (Fig. 2).

Figure 1: the three components of the loop. The Assistant advises, the Human translates advice into commands and executes them, and the environment returns fresh first-person frames and state; turns repeat until the Assistant declares done.

Physical layer: a 156-D pose. Each frame is parameterized by a 156-dimensional axis-angle vector covering root orientation, 21 body joints (torso, head, arms, legs) and 15 articulated joints per hand; root translation is stored separately in world coordinates, and a fixed 90 degree rotation about x maps the y-up model onto the z-up world. Standard SMPL+H forward kinematics and linear blend skinning yield joint transforms and deformed vertices; the body mesh is authored once at /World/Body and each frame updates only vertex positions and normals, so topology, material binding and camera attachment never change.

$$\mathbf{q}=[\bm{\omega}_{\mathrm{root}},\bm{\theta}_{\mathrm{body}},\bm{\theta}_{\mathrm{left}},\bm{\theta}_{\mathrm{right}}]\in\mathbb{R}^{3+63+45+45}$$

In the released code this equation lives in simulator/human_model/smplh_fpv.py: the right-hand block starts at RHAND_LO = 37*3 = 111, so 66 to 111 is the left hand and 111 to 156 the right, matching the paper's 45+45 split exactly. Reading the code confirms the dimensions are not a typo, which is rarer than it should be in human-model papers.

Action layer: collision-aware navigation. Routes are planned on a 0.10 m ground grid rasterized from the same collision world, with a cost function that adds two terms to the eight-neighborhood step length: one that biases paths toward open space and one that charges for closed doors.

$$c(n,n^{\prime})=r\left[\ell_{8}(n,n^{\prime})+\lambda\max\bigl(0,d_{\mathrm{safe}}-D(n^{\prime})\bigr)+c_{\mathrm{door}}(n^{\prime})\right]$$

Here r = 0.10 m is the grid resolution, the step length takes 1 or the square root of 2, lambda is 8 and d_safe = 0.35 m. The clearance term is a preference, not a hard obstacle: narrow passages stay passable. For a closed door the planner identifies the blocking leaf, routes to a collision-validated handle stance, executes the opening action, updates occupancy and replans; if no such sequence exists the action returns PATH_BLOCKED instead of teleporting the human. The repository's nav_planner.py implements this cost in _cell_step_cost, with the clearance margin and weight exposed as runtime parameters (defaults 0.35 and 2.0); the paper's lambda = 8 is an experiment configuration, not a hardcoded constant.

Action layer: target-driven manipulation. Grasp selection first filters candidates by a cylindrical rule: an object whose height exceeds 1.15 times its largest cross-section diameter is grasped from the side, otherwise from above. Arm inverse kinematics, reach and tool trajectories follow, and hand posture adapts to object size. The line geom["h"] > 1.15*max(cross_diam, ...) in grasp_select.py is precisely this rule. Transforming objects into the body frame costs one homogeneous inverse:

$${}^{W}\!T_{O}=\left({}^{\mathrm{world}}\!T_{W}\right)^{-1}{}^{\mathrm{world}}\!T_{O}$$

Semantic layer: 36 primitives and a deterministic grammar. Commands state only the action and its targets, for example [Grab] cup or [PutInside] pizza oven; navigation, stance and body motion are filled in by the action layer. The grammar is deterministic:

$$\texttt{[Verb] <entity\_name> (instance\_id) [<entity\_name> (instance\_id) ...]}$$

The repository's command_grammar.py keeps a verb registry and auto-inserts a navigate step for a bare [Grab]: the human must first walk within reach, and neither the assistant nor the scorer should have to remember that. The 36 primitives span navigation, observation, manipulation and daily activity, from [Walk], [LookAt], [Grab], [Put], [PutInside], [Open] and [TurnOn] to [Pour], [Cook] and [Cut].

The closed-loop turn protocol. Each turn the Assistant receives the first-person frames so far, the Human's previous utterance and the task description, and emits interrupt with natural-language guidance, silent, or done. The Human then emits structured commands plus a natural-language reply; the environment executes them and returns new keyframes, reward signals and events. Keyframes sampled during an action join the visual input, capped at 32 frames per turn with 4 frames per action; over the cap the 8 most recent frames are kept and 24 are sampled uniformly from older history.

$$(I_{\leq t},u^{H}_{t-1},\tau_{t})\xrightarrow{\text{assistant}}u^{A}_{t}\xrightarrow{\text{human}}(c_{t},u^{H}_{t})\xrightarrow{\text{environment}}(I_{t+1},r_{t},e_{t})$$

Tasks and scoring. The 120 tasks fall into six categories: Guidance, Planning, Accessibility, Decision, Prevention and Finding. Each task opens with a warmup in which the human performs an authored prologue (looking around, reading a label) so the assistant receives background evidence before the loop starts. Each task predefines a set of core states, such as the pizza is in the oven or the fridge is closed; when the episode ends the simulator's actual states are checked and the task passes only if every required state holds. With 20 instances per category,

$$\operatorname{PassRate}(F)=\frac{1}{20}\sum_{t\in F}s_{t},\qquad\operatorname{MCR}(F)=\frac{1}{20}\sum_{t\in F}c_{t}$$

MCR is the mean fraction of objectives completed, not an action ratio. Nine capabilities (object recognition, spatial grounding, OCR, long-horizon memory, planning, knowledge, error detection, proactivity, communication) each carry their own state-verifiable check points, and a capability score is simply the pass rate over those points:

$$\operatorname{CapScore}(k)=100\times\frac{\sum_{t\in E_{k}}p_{t,k}}{\sum_{t\in E_{k}}n_{t,k}}$$

The analysis also uses a normalized horizon: each task's step budget is min-max normalized within its family so tasks from different categories sit on one common short-to-long axis,

$$h_{t}=\frac{B_{t}-\min_{j\in\mathcal{F}(t)}B_{j}}{\max_{j\in\mathcal{F}(t)}B_{j}-\min_{j\in\mathcal{F}(t)}B_{j}}$$

flowchart TD
  A["Assistant MLLM: first-person frames + dialogue history"] -->|"interrupt / silent / done"| H["Human MLLM"]
  H -->|"grammar-exact commands, one per line"| SEM["Semantic layer: 36 primitives, bare verb auto-navigates"]
  SEM --> NAV["Action layer: 0.10 m grid A* with clearance cost"]
  SEM --> MAN["Action layer: grasp selection + arm IK + hand adaptation"]
  NAV --> PHY["Physical layer: SMPL+H 156-D kinematic drive, 17-capsule proxy"]
  MAN --> PHY
  PHY --> CAM["Head-camera keyframes + true environment state"]
  CAM --> A
  CAM --> CHK["Termination check: pass only if all core states hold"]
  CHK -->|"otherwise MCR records partial progress"| A

Figure 2: the loop redrawn from the paper's three layers and turn protocol. Verdicts come from environment state, never from the model's own claims.

Figure 3: system overview. Physical, human-action and semantic layers stack bottom-up, with the semantic layer facing the two-MLLM turn protocol.

Experiments

Headline: the best model finishes four tasks in ten. Claude Fable 5.1 leads the twelve models with a 41.7% pass rate and 54.8% MCR; every other model stays at or below 33.3%. By category, the best model reaches 85.0% on Decision but only 20.0% on Guidance and 25.0% on Planning: tasks that demand long-horizon tracking and multi-step advice are exactly where current models collapse. The same Human with an Oracle Assistant scores 100% and with no assistance 0%, so the 41.7% is not a statement about task difficulty but about unreliable advice.

Table 1: pass rates from the paper's Table 2 (category columns are Pass %). Oracle 100% and no-help 0% are the two endpoints of this scale.

Averages do not decide outcomes; the weakest link does. GLM-5.3-Flash and Claude Fable 5.1 have nearly identical nine-capability averages (58.2 versus 59.2), yet GLM scores markedly lower on communication (69.8 versus 79.6) and completes 11.7 percentage points fewer tasks (30.0 versus 41.7). The error analysis points the same way: even for Fable, 13.6% of first errors are R5, guidance that cannot be executed in the current situation. Closed-loop success is bounded by the weakest capability in the chain rather than lifted by the average, and this kind of communication, turning a goal into words a human can act on, is exactly what static question-answering benchmarks never measure.

Table 2: excerpt of the paper's Table 3 (check-point pass rates, %). The gap between the communication column and the average is the direct evidence for the weakest-link claim.

Efficiency: every redundant sentence is someone's legs. Measured against the Oracle's action cost on tasks each model passes, nearly all models show severe redundancy, peaking at 2.35x to 3.66x the Oracle on Accessibility and Guidance. Efficiency is task-dependent rather than a fixed model trait: Fable is the least efficient on Accessibility (6.75x) yet among the best on Prevention (1.20x), while Kimi K3 matches the Oracle on Finding (1.00x) but is the most expensive on Guidance (3.66x). Unnecessary instructions translate directly into wasted human effort, which makes interaction efficiency as important as success rate.

Memory augmentation helps, but not enough. Appending a hidden Memory field to the Assistant's response, letting it record events for later turns, improves success by 5 to 15 percentage points on Prevention and Finding across all three tested models. Effects on Decision, Guidance and Accessibility vary by model, and Planning drops by 5 points for two of them. A simple inference-time memory patch does not buy reliable long-horizon assistance; the authors accordingly point toward trained native memory mechanisms.

Table 3: excerpt of the memory-augmentation ablation. The gains on Prevention and Finding are stable; the Planning regression is equally real.

Longer is harder, and failures concentrate in perception and memory. Sorting tasks along the normalized horizon axis, the family-adjusted slope is -0.431: success falls 43.1 percentage points from the shortest to the longest tasks (Figs. 4 and 5). Errors fall into six classes: R1 misreads a visible object or location, R2 loses track of what happened or ignores an action result, R3 applies the wrong rule or misses a consequence, R4 builds or updates the wrong multi-step plan, R5 issues guidance that cannot be executed now, R6 speaks at the wrong time or stays silent when help is needed. Across models, perception and memory together (R1+R2) account for 70.4% to 85.7% of first errors: in a closed loop, seeing and remembering break first.

Figure 4: error cases from the paper's Fig. 4. R5, unexecutable guidance, still accounts for 13.6% of first errors even for the strongest model.

Figure 5: success rate against normalized horizon; the family-adjusted slope is -0.431.

How trustworthy is the proxy human? Using an MLLM as the Human is the largest methodological risk of the protocol, and the authors run a human-human control: agreement between the agent and human outcomes is 84.3%, while human-human agreement is 85.2%. The proxy behaves close to people on this benchmark, though the authors are explicit that close is not identical.

Limitations

The human is kinematic, not physical (authors' own). The body accepts no forces: it cannot fall, be shoved, or be blocked by a swinging door. That trade buys reproducibility and gives up contact forces, pushing and imbalance, an entire family of real interactions; advice like push harder has no referent in this world.

The Human agent is not a real user (authors' own). The D.5 agreement study shows the proxy's outcome distribution is close to humans', but real people mishear, improvise and refuse instructions, none of which the protocol models; the benchmark measures assistance to a compliant, capable human.

Horizon is correlation, not causation (authors' own). The -0.431 slope describes that longer tasks are harder; it must not be read as shortening tasks would raise success. Longer tasks also carry more objects, more turns and more memory load, and those variables are not separated.

Simple memory augmentation is insufficient (authors' own). The hidden Memory field yields unstable gains and a negative effect on Planning, indicating a structural gap between inference-time patches and long-horizon assistance.

Our additional reading. First, the main experiments fix the Human to Gemini 3.1 Pro, which suppresses human-side variance and cleans up model comparisons, but in deployment the variance of the human is part of the difficulty of assisting. Second, 44 admitted scenes and 120 tasks support directional conclusions, not fine-grained claims about generalization across scenes. Third, state-based scoring requires tasks whose success can be written as a set of core states, so assistance that cannot be state-checked, comforting a frustrated person for instance, is outside the benchmark by construction; that boundary is also its scope.

Conclusion and Outlook

InteractiveWorld's contribution is not a model or an algorithm but an object of measurement: it makes assistance reproducible and scoreable. The three-layer architecture settles the three engineering prerequisites of stability, naturalness and executability; the two-MLLM loop with Oracle and no-help endpoints fixes the scale; and InteractiveBench's 120 tasks with nine capability check lists localize failure to specific links of the chain. The readings are sobering: 41.7% for the best model, a -0.431 slope along horizon, and over seventy percent of first errors in perception and memory. The environment also draws a research path: trained native memory for dynamic interaction, communication optimized as a first-class capability, and interaction efficiency inside the objective. For anyone building embodied assistants, its greatest value is simple: for the first time you can ask whether your model actually helped a person, and get an answer the model did not write itself.

Golden Sentence

"Closed-loop success is bounded by the weakest capability in the chain rather than lifted by the average, and the communication that turns a goal into words a human can act on is exactly what static benchmarks never measure."