PAPER DEEP DIVE
On Agentic Manipulation Policies
Generalist AI models can control manipulators through persistent Python code without demonstrations or task-specific training. AMP1 reaches 85.6% zero-shot on RoboLab-120 and 104 of 108 real-world trials; persistent memory raises RoboDojo success from 26.4% to 84.1%.
One-Sentence Summary
The paper defines agentic manipulation policies as systems that keep a generalist model in the robot control loop, letting it inspect the scene, write persistent Python code, and directly revise its actions; its AMP1 reference harness reaches 85.6% zero-shot success on RoboLab-120 and, with a persistent memory store, raises RoboDojo success from 26.4% to 84.1% without demonstrations or weight updates.
Why Keep the Model in the Loop?
Most contemporary manipulation systems compress robot competence into a specialized policy. Vision-language-action models map observations and instructions to actions, while world-action models learn dynamics or future states that support control. These systems can be fast and can fit narrow distributions extremely well. Their weakness appears when the scene falls outside the training distribution: an object is too wide for the gripper, a visible target is hidden under another object, a camera is miscalibrated, or a contact-rich motion fails halfway through. A dedicated policy may continue executing a familiar pattern because it has little mechanism for reconsidering the meaning of the new evidence.
Skill orchestration offers one alternative. A generalist model chooses among low-level primitives such as grasp, push, and place. This changes the task plan when observations change, but the model's authority ends at the primitive boundary. If a grasp planner cannot find a grasp for a flat book, the model can request another grasp, yet it cannot easily decide to push the book to the table edge first. The available interface controls the granularity of the model's intelligence.
Offline program synthesis offers another alternative. A model writes a program before deployment, and the program reads sensors and reacts while it runs. This provides more freedom than a fixed skill menu, but the model that authored the program is no longer present to revise its assumptions. The runtime can execute feedback rules, but it cannot invent a new global strategy when the original representation of the world is wrong.
The paper instead studies agentic manipulation policies, or AMPs. A generalist model remains active throughout the episode. It can request images, query robot state, reason about contact geometry, write executable code, preview a trajectory, observe execution, and then decide what to do next. The distinction is not merely a larger number of model calls. It changes the policy representation from a fixed neural mapping or a prewritten script into an evolving executable strategy whose author is still available.
This route became plausible because frontier models have improved at spatial reasoning, tool use, programming, and long-horizon problem solving. The paper asks an engineering question that follows from those changes: if sensing, code execution, scene representation, motion planning, and robot feedback are exposed through a disciplined harness, can one generalist system control single-arm, bimanual, and mobile manipulators without demonstrations or task-specific fine-tuning?
From a Policy Network to an Executable Robot Agent
AMP1 separates the generalist model from the robot through a harness. The model writes Python that calls an SDK. A robot server answers sensor requests, maintains a perceived scene, validates motion, and executes trajectories. A route policy limits which calls are available, and reference documentation is appended to the model prompt. The instruction, robot model, remaining control steps, and sensor observations all enter through this interface.
The basic harness deliberately omits grasp, push, and place skills. It exposes camera images, depth, calibration, proprioception, segmented objects, point clouds, collision checks, timed end-effector waypoints, and gripper commands. This design gives the model low-level control authority. It also preserves detailed failure information: a blocked arm, a rejected pose, a weak grasp, or a collision remains visible to the same intelligence that selected the strategy.
The model can therefore use the harness in two modes that are often separated in robotics. It reasons about the task in language, then implements its geometric decisions in code. A single turn may retrieve an object's points, fit an oriented box, choose a grasp direction, generate an approach path, test reachability, preview the motion, inspect collisions, execute the chosen segment, and display a wrist-camera image for the next turn.
Inside AMP1
Persistent Python as an Episode Workspace
Every episode starts one sandboxed Python kernel that persists across model turns. Imports, arrays, variables, helper functions, and any other in-memory state remain available until the episode ends. The kernel returns printed output, tracebacks, displayed images, remaining control steps, and wall-clock time. The next model call reads those results and continues from them.
This persistence matters because manipulation is stateful. A model may compute a table plane in one turn, use it to correct a grasp in the next, and reuse a coordinate transform after a failed motion. In zero-shot operation, the kernel is empty at the beginning of every episode, so the model writes its search and geometry routines anew. The persistent state is within an episode, not across the full task dataset.
The public code maps directly to this design. src/amp/kernel/kernel.py describes a persistent, sandboxed Python kernel per episode that holds the harness SDK. src/amp/kernel/child.py executes model code in the same namespace and returns a structured result to the dialogue. The implementation confirms that the harness is an executable workspace, not a wrapper that merely translates natural-language tool calls into fixed robot actions.
An Editable Scene Model
The sensing layer returns RGB, depth, camera intrinsics, camera poses, robot joint state, end-effector state, and gripper state. A colored point cloud can be fused across cameras. The server voxelizes this cloud and groups connected voxels into candidate objects, stopping at sharp color boundaries or places where objects touch. Small fragments are merged or removed. Each retained cluster is extruded downward into a closed mesh, which becomes an obstacle region for collision checking.
Object identity is maintained across updates through a minimum-cost assignment. A previous object and a new detection can match only when their centroids are close enough and their colors and sizes agree within tolerances. A displaced object receives a new identifier, while a temporarily occluded object may retain its reserved identifier for a fixed period. Each update reports additions, removals, and changes so the model can follow how its representation evolved.
The model is allowed to edit the scene. It can grow a region from one pixel, define a box from a point cloud, replace a wrong mesh, or delete a false object. This is important because segmentation is not treated as ground truth. The prompt warns that objects may be merged, split, or filled incorrectly, and instructs the agent to inspect the perceived model against images and repair it before planning. The world model is thus part of the policy, not a passive preprocessing stage.
Timed TCP Waypoints and Verified Motion
The motion interface accepts timed tool-center-point poses and gripper commands. A waypoint is:
$$w_i=(T_i,t_i,g_i),\qquad T_i=(p_i,R_i)\in SE(3),\qquad g_i\in\{0,1\}$$
Here pi is position, Ri is orientation, ti is arrival time, and gi is the open or closed gripper command. The model creates these poses relative to measured references such as the current TCP pose, an object centroid, or a previous waypoint. A bimanual robot receives a waypoint sequence for each end effector.
For a segment, the controller computes normalized time:
$$u=\frac{t-t_{i-1}}{t_i-t_{i-1}}$$
Position and orientation are interpolated separately:
$$p(t)=p_{i-1}+\sigma(u)(p_i-p_{i-1})$$
$$R(t)=R_{i-1}\exp\!\left(\sigma(u)\log(R_{i-1}^{\top}R_i)\right)$$
The quintic time scaling σ reduces jerk. At every control step of duration Δt, the system solves inverse kinematics using the previous joint configuration as a seed:
$$q_k=\tau(k\Delta t)$$
The resulting configuration is checked against the interpolated pose, joint velocity limits, and optional collision samples. The scene's obstacle region is:
$$\mathcal{O}=\bigcup_j M_j$$
Let A(q) be the volume occupied by the robot at configuration q. Colliding and free configurations are defined as:
$$\mathcal{C}_{\mathrm{obs}}=\{q\in\mathcal{C}\mid\mathcal{A}(q)\cap\mathcal{O}\neq\emptyset\}$$
$$\mathcal{C}_{\mathrm{free}}=\mathcal{C}\setminus\mathcal{C}_{\mathrm{obs}}$$
If a trajectory is unreachable, too fast, or colliding, the server returns the reason and the affected steps. The model can move the goal, slow down, rotate the gripper, or choose another approach. This repair loop is central to the method rather than a peripheral convenience. In 1,200 RoboLab episodes, AMP1 repaired a refused plan after reading the harness reason in 40% of episodes and diagnosed an execution failure before replanning in 39%.
The code path is visible in src/amp/robot_server/api/motion.py, where preview, execute, gripper, hold, home, base motion, and stop are exposed separately. src/amp/robot_server/world.py maintains the perceived scene and the held object used by collision and assist planning. The separation between preview and execution lets the model inspect a plan without committing the physical robot.
Direct Control Versus Assistive Routines
The paper compares the basic harness with an assistive variant. The basic interface gives the model waypoints, gripper commands, motion preview, and collision tools. The assistive interface adds ranked grasp and placement candidates plus routines that execute, verify, and retry a grasp automatically. The model still selects targets, orders subtasks, and decides when the task is complete.
The experiment reveals an interaction between model strength and interface abstraction. The four strongest models all perform worse with the assistive interface, by as much as 7.5 percentage points. The six weakest models improve, with the weakest gaining 17.6 points. Assistive routines help when their internal grasp success exceeds the model's own control skill, but they hide failure details from models capable of diagnosing those details. Direct control is more burdensome, yet it gives a strong model the observations needed to repair the action.
flowchart LR
T[Language instruction] --> M[Generalist model]
M --> K[Persistent Python kernel]
K --> S[Sensing and scene SDK]
S --> P[Point cloud and editable meshes]
K --> W[Timed TCP waypoints]
W --> C[IK and collision preview]
C --> R[Robot server execution]
R --> F[Robot state and camera feedback]
F --> M
P --> M
C -->|rejection reason| M
AMP1+ and Gradient-Free Memory
AMP1+ adds a persistent workspace M that survives across episodes. The system imposes no schema: the workspace can contain Markdown notes, Python libraries, parameters, data, or image indices. The agent reads it at the start of an episode and decides what to retain after practice. Model weights remain fixed. Experience is represented as editable external knowledge rather than a gradient update.
In few-shot practice, a batch of layouts shares the current workspace. After the batch, a reflecting instance sees rewards and transcripts and produces the next memory version:
$$\mathcal{M}_{k+1}=F\!\left(\mathcal{M}_k,r_{0:B}^{0:k},\tau_{0:B}^{0:k}\right)$$
The paper uses batches of four and a budget of 48 practice episodes, allowing up to 12 rounds. The execution agent sees only the current workspace, while the reflecting agent sees all previous rewards, code, outputs, and camera records. The final workspace is frozen for testing on unseen layouts.
Continual learning uses one shared workspace and visits the 42 RoboDojo tasks in a random order. After one episode and its binary result, the agent updates the external state:
$$\mathcal{M}_t=G\!\left(\mathcal{M}_{t-1},r_t,\tau_t\right)$$
This formulation tests both retention and transfer. If the agent only memorized task-specific coordinates, later tasks would gain little. The paper instead observes transfer: after completing the first 10-task segment, success on the unseen fourth segment rises from 24% to 47%. Memories that help across tasks include robot frames, physical limits, methods for triangulating without depth, reusable grasping code, and rules about when to open the gripper.
flowchart TD
E[Practice episode] --> R[Official binary reward]
E --> X[Code messages outputs and images]
R --> D[Debrief and reflection]
X --> D
D --> N[Markdown notes]
D --> C[Reusable Python library]
D --> P[Parameters and failure conditions]
N --> M[Workspace for later episodes]
C --> M
P --> M
M --> I[Read import and call]
I --> E
Evidence
Zero-Shot Simulation Results
The evaluation spans RoboLab-120 and three CaP-Gym suites. RoboLab contains 120 language-specified manipulation tasks with default, vague, and specific instructions. LIBERO-PRO perturbs object positions and instructions. robosuite covers five single-arm and two bimanual tasks. BEHAVIOR-1K contributes two mobile-manipulation tasks. The same harness is used throughout; only the robot description and camera configuration change. The paper runs 10 episodes per RoboLab task for each instruction style, totaling 3,600 episodes, plus 3,000 LIBERO-PRO, 700 robosuite, and 50 BEHAVIOR-1K episodes.
With Gemini 4 Argon, AMP1 reaches 85.6% on RoboLab-120 under default instructions, with a 95% confidence interval of 83.5% to 87.5%. Opus 5.5 reaches 82.8%. At the time of writing, the strongest public leaderboard agent, SenseNova-RoboRSI, reaches 57.1%; the strongest learned policy, Lakeside, reaches 51.0%; the strongest VLA and WAM entries reach 49.2% and 42.9%. These comparisons are all zero-shot with respect to demonstrations and task-specific training, although both the policy and the harness are informed by general pretraining.
Instruction robustness is as notable as the peak score. Gemini receives 84.0% under vague instructions and 87.1% under specific instructions, while Opus 5.5 receives 79.8% and 82.8%. The spread is about three points. Specialized policies are more brittle: FLUX 3 Action falls from 42.9% to 28.4% under vague wording, and π0.5 falls from 28.0% to 15.2%. A generalist model can reinterpret an underspecified request instead of treating wording as a distribution shift.
| Benchmark | Scale | AMP1 | Reported comparison |
|---|---|---|---|
| RoboLab-120 | 120 tasks, 10 trials each, three instruction conditions | Gemini 4 Argon: 85.6%; Opus 5.5: 82.8% | Best VLA: 49.2%; best WAM: 42.9%; leading agent: 57.1% |
| LIBERO-PRO | 60 perturbed tasks, 50 trials each | 95.2% | Zero-shot CaP-Agent0: 18.2%; fine-tuned π0.5: 34.9%; HarnessPAI: 96.5% |
| robosuite | 7 tasks, 100 trials each, including two bimanual tasks | 700/700, 100.0% | HarnessPAI: 96.3%; human programs: 82.6% |
| BEHAVIOR-1K | 2 mobile tasks, 25 trials each | 48/50, 96.0% | CaP-Agent0: 64%; human programs: 54% |
The LIBERO-PRO comparison requires context. HarnessPAI reaches 96.5%, just above AMP1's 95.2%, but it improves its program over rollouts of the evaluated tasks and is not zero-shot. Its advantage comes entirely from LIBERO-Goal; AMP1 is ahead on LIBERO-Object and LIBERO-Spatial. Many AMP1 failures in LIBERO-Goal occur because the agent believes it has succeeded under a reasonable interpretation of the instruction, while the benchmark verifier uses an unstated convention about the front of a stove or the required drawer opening.
Physical Robot Trials
The real-world study uses 12 tasks and three DROID-like platform instances. Each method receives three trials per task on each instance, producing nine trials per task and 108 per method. All instances use a seven-degree-of-freedom Franka Research 3, a Robotiq 2F-85 gripper, and ZED cameras, but they differ in tabletop appearance, camera calibration, cabling, and hardware wear. A scene restoration tool aligns initial object poses across methods. AMP1 with Opus 5.5 succeeds in 104 of 108 trials, or 96.3%. FLUX 3 Action succeeds in 49, and π0.5 in 30.
The tasks include exclusion, color and category grounding, ordering, memory, non-prehensile actions, and enabling actions that create a grasp. For AMP1, the average time over successful trials is 280 seconds: 69 seconds of execution and 211 seconds of reasoning. The decomposition is important. The system is not claiming real-time control. It is claiming that broader competence can compensate for current inference latency, while explicitly identifying speed as the next barrier.
| Configuration | Success | Rate | Protocol detail |
|---|---|---|---|
| AMP1 with Opus 5.5, recovery enabled | 104 / 108 | 96.3% | 12 tasks, 3 hardware instances, 3 trials per task and instance |
| AMP1 with Opus 5.5, recovery disabled | 94 / 108 | 87.0% | A controller error that interrupts motion terminates the trial |
| FLUX 3 Action | 49 / 108 | 45.4% | Specialist WAM baseline |
| π0.5 | 30 / 108 | 27.8% | Specialist VLA baseline |
Error recovery accounts for 10 additional successes. AMP1 can query the arm state and request recovery after Franka's collision reflex interrupts motion. The gain is largest when the gripper or a held object operates near the scene: packing canned foods rises from 3/9 without recovery to 9/9 with it. This is a concrete example of agentic control. The model does not merely choose the next target; it interprets an unexpected controller event, recovers the robot, and constructs a safer continuation.
Emergent Manipulation
The paper documents behaviors that were not supplied as primitives. A flat book is pushed to the table edge so that a graspable overhang appears. An occluding object is moved to reveal a target. A fruit too wide for the gripper is pinned against a heavier object and scooped from below. A long object is lowered by resting one end first and sweeping the other. A blocked motion reveals an unseen shelf; the agent infers where the shelf must be and changes its approach. On the physical robot, discrepancies between commanded and observed gripper motion are used to estimate and compensate for calibration error.
These examples are not treated as proof that the model understands physics in a human sense. The authors quantify recurrence across 1,200 RoboLab episodes by matching executed code and stated intentions against skill patterns. A pattern counts only if it appears in at least five episodes and at least two tasks. Measuring a container before placement appears in 66% of episodes, aligning a grasp with object axes in 61%, reasoning about long and short axes in 47%, repairing a refused plan in 40%, and diagnosing an execution failure before replanning in 39%. The statistics support a modest claim: the model repeatedly composes reusable manipulation routines from the interface rather than relying on one-off lucky sequences.
Learning Without Weight Updates
RoboDojo provides 42 bimanual tasks and only RGB input, with no depth or camera calibration. Zero-shot AMP1 succeeds in 26.4% of unseen layouts. AMP1+ practices each task on seed-0 layouts for at most 48 episodes and is tested on seed 2. Its success rate rises to 84.1%, above the strongest leaderboard entry at the time, Q0-Preview at 68.9%. It also uses about 35% fewer model calls. By round two, 39 of 42 task workspaces contain a code library, so later episodes import procedures instead of rediscovering them.
The Mahjong example explains the mechanism. The robot must place a tile upright at the right end of a row, while the hidden score also requires the tile's face to have the correct orientation. At round two, the reflection agent compares a failed and a successful episode. It enlarges the final frames, notices that the red symbol is at the bottom in the successful episode and at the top in the failed one, traces the grasp to opposite ends of the face-down tile, and records which end is the top. A later round also learns to open the gripper before reaching into the stack. Held-out success for the task rises from 10% before practice to 100%.
Continual learning uses one workspace across a random permutation of all 42 tasks. One pass raises aggregate success from 26.4% to 57.6%. After the first ten tasks, the unseen fourth segment improves from 24% to 47%, while earlier segments retain much of their gain. The workspace is therefore not a lookup table of exact layouts. It accumulates embodiment knowledge and executable primitives that transfer to later scenes.
Controlled Experiments
Holding the harness fixed and varying 13 models produces success rates from 12.8% for Qwen3.6-35B-A3B to 85.6% for Gemini 4 Argon. Within the Claude family, Opus 5.5 reaches 82.8%, Haiku 5.5 reaches 51.6%, and the older Opus 4.5 reaches 35.2%. The highest open-weight result is Qwen3.8-Flash-Next at 64.9%. More reasoning improves every evaluated model, with Claude models gaining as much as 17.2 points. Model capability is the largest single factor in these experiments.
The sensory ablations show a different pattern. Opus 5.5 retains 81.9% without perception tools, 81.1% without depth, and 75.8% with RGB alone. Its mean model calls rise from 34.8 to 38.7, 46.3, and 53.5. When depth disappears, the agent matches pixels across views and triangulates points. When calibration disappears, it can move the wrist camera through known poses, match SIFT features, and estimate intrinsics and camera-to-gripper transforms by least squares. Strong models convert missing sensing into extra computation and active movement.
Weaker models cannot perform the same recovery. Haiku 5.5 falls from 51.6% to 10.7% when only RGB remains. Replacing images with textual descriptions reduces Opus 5.5 from 82.8% to 36.8%, contrary to prior work that advocated a textual scene representation. Camera count is less important than geometric content: with full sensing, all cameras yield 82.8%, scene cameras alone yield 80.8%, and the wrist camera alone yields 80.0%. In the wrist-only case, the model withdraws the arm to bring more of the table into view.
| Input | RoboLab-120 success | Mean calls per episode | Agent response |
|---|---|---|---|
| RGB + calibration + depth + perception tools | 82.8% | 34.8 | Full harness input |
| RGB + calibration + depth | 81.9% | 38.7 | Detects objects and lifts them to 3D |
| RGB + calibration | 81.1% | 46.3 | Triangulates matching points across views |
| RGB only | 75.8% | 53.5 | Uses active views to recover geometry and calibration |
| Text descriptions only | 36.8% | Reported separately in the figure | Loses spatial detail needed for control |
Failure Analysis and Cost
Gemini with AMP1 fails 173 of 1,200 default RoboLab trials. Of those failures, 121 are benchmark artifacts: objects whose geometry or mass defeats a standard parallel-jaw grasp, or verifiers that reject an end state satisfying the instruction. Opus 5.5 has 206 failures, 135 of which are such artifacts. Across the 95 tasks without these defects, Gemini succeeds in 914 of 950 trials, or 96.2%. The paper separates these cases instead of attributing every failed episode to the policy.
The cost analysis favors trajectory-level agentic control. For Opus 5.5, an average RoboLab episode costs $1.39 at list prices. The paper reports this as about 26 times lower than Astranomy with GPT-6 Astra at $36.40, while AMP1 also has higher success. The saving comes from issuing a whole trajectory in one model turn instead of requesting every small end-effector step, and from making images enter context only when the model asks to view them.
Limitations
The authors identify two primary limitations. First, inference is slow: the robot spends much of an episode waiting for the model to reason. Second, the model cannot reconsider a trajectory while that trajectory executes. The evaluation focuses mostly on quasi-static scenes, whereas a moving target or slipping object may require feedback before the next model turn. These are not minor implementation details; they define the present operating regime.
The architecture also depends on remote model providers in many experiments. Network failures, rate limits, changing model versions, and pricing can affect behavior or reproducibility. Errors in the paper include provider-side blocks. The public harness reduces some integration risk by keeping the robot interface explicit, but it does not eliminate reliance on a service outside the robot.
Physical evidence covers one arm and gripper family across three DROID instances. Bimanual and mobile manipulation are demonstrated in simulation, which is valuable but not the same as long-duration physical deployment. The claim of cross-embodiment generality should therefore be read as strong initial evidence across the tested platforms, not as proof that every robot configuration will work without adaptation.
Perception remains a dependency. Depth, calibration, and scene segmentation can be repaired by a strong model, but at higher latency and model-call cost. Weak models often fail outright when geometric sensing is removed. Benchmark verifier mismatches show that reported success is conditional on an evaluation protocol. Memory adds another risk: an accidentally successful strategy may be generalized incorrectly, and the workspace has no described conflict audit or rollback process. Executable memory also needs provenance and security review in a multi-user deployment.
Safety is only partly addressed. The model previews trajectories, the server checks reachability, velocity, and sampled collisions, and the physical platform retains protective reflexes. These mechanisms are useful engineering safeguards, but they are not a formal guarantee that generated code will always command a safe action.
Code and Reproducibility Notes
The public repository contains the reference harness and paper checkpoints. The persistent kernel is implemented under src/amp/kernel/. Motion routes are exposed in src/amp/robot_server/api/motion.py, while src/amp/robot_server/world.py maintains the scene and held object. Assistive grasp and place calls live in src/amp/robot_server/api/assist.py.
docs/practice.md defines AMP1+ as AMP1 with a workspace that is read as ./workspace in later episodes. Few-shot practice stores one workspace per task, while continual practice shares one workspace across all 42 tasks. Evaluation is read-only. The repository includes fewshot_best5 and continual_chain00 checkpoints so the paper's learning states can be evaluated without rerunning all practice episodes.
The practice protocol uses batches of four, at most 48 practice episodes per task, and evaluations on unseen seed-2 layouts. Debriefs receive official outcomes and episode steps, use a separate Python kernel without robot access, and write through a workspace guard. These implementation details make the “learning” claim precise: the model changes external notes and code, not its neural weights.
Conclusion
The paper's central contribution is a reframing of manipulation control. Instead of asking a specialist network to absorb every recovery behavior from demonstrations, AMP1 exposes a robot interface that a generalist model can program and revise during execution. Persistent code, editable scene geometry, timed TCP trajectories, previews, collision feedback, and direct robot observations form a coherent closed loop. The same harness succeeds across four simulation suites and 104 of 108 physical trials without demonstrations or task-specific training.
AMP1+ adds a practical form of experience without gradient updates. A workspace of notes and executable functions improves unseen-layout success on RoboDojo and transfers some knowledge to tasks that have not yet been attempted. The result is not a deployable real-time controller. It is evidence that agentic manipulation can trade inference time for breadth and robustness, and that this tradeoff may be improved by reusing code, distilling experience into faster controllers, and overlapping planning with execution.
The more durable lesson is that capability and speed should be evaluated together. A fast policy that fails half the tasks is not a usable general solution, while a slow policy that succeeds can become practical as inference costs fall and local feedback improves. The paper makes that tradeoff explicit and turns it into a concrete systems research agenda.
Key Takeaway
The defining property of an agentic manipulation policy is not that a language model emits robot commands. It is that the model remains able to revise the control strategy, using its own observations and failures as part of the policy loop.
