Gemini 4 Argon 面向机器人：贯穿控制、Real2Sim 与自我改进的物理推理模型

Gemini 4 Argon 在物理理解、代码生成与视觉-语义推理上实现了一次阶跃式进步——它打通了机器人技术栈上的一系列新工作流，并加速通往物理 AGI 的进程。

这一能力跃迁可以在 Embodied Reasoning（ERQA）基准上看到：该基准测试模型在 3D 环境中理解并求解物理与空间问题的能力。仅凭纯推理、不借助任何工具，Argon 取得 87% 的成绩，超过所有其他前沿模型。

作为通用物理推理模型，Argon 可以覆盖机器人生命周期的各个环节。本文给出 Argon 在四个关键方向上的代表性提示词：

• 机器人控制 → 结合 MuJoCo 练习的移动操作、零样本机器人控制、以及对视频世界模型的闭环引导。
• Real2Sim → 从机器人相机画面、第一人称人类视频或手绘纸面草图，重建物理标定过的仿真孪生。
• 物理理解 → 智能体式多模态事后标注，包括对灵巧性的细粒度理解、稠密价值函数与进度估计。
• 研究与自我改进 → 借助仿真改进动作，以及在真机硬件上的迭代自我改进。

用 Argon 解锁机器人控制

1. Spot 移动操作：踩脚踏垃圾桶丢易拉罐

Argon 控制一台 Boston Dynamics Spot® 四足机器人（12 自由度腿部 + 6 自由度机械臂）学习一项新技能：踩下脚踏式垃圾桶的踏板，把易拉罐丢进去。Argon 先为脚踏垃圾桶建一个带关节的 MuJoCo 数字孪生，在仿真里练习动作，然后再去踩真实的踏板、把易拉罐丢进敞开的桶里。

提示词：

There's a Diet Coke can behind you and another on the black table on your left.
Use the spot-mobile-manipulation skill to pick them up and trash them in the black foot-pedal bin.
Before stepping down on the pedal, verify in the front cameras that your front-right foot target is centered on the pedal, and use a 2-agent verification swarm (foot-verifier and hand-verifier) to verify both the foot-pedal step and the 3D hand-in-bin alignment before releasing.

2. 零样本控制

Argon 控制一条简单的机械臂，把倒下的马克杯扶正，并把揉皱的纸巾放进杯子里。Argon 借助视觉坐标参照系与动作原语，经过多个回合的推理完成该任务。

提示词：

[Attached: wrist_cam.png, obs.json]

Task: Get the robot arm to reorient the mug upright and put the napkin inside the mug.

## Execution Discipline
- Simply output 1 sentence observation followed immediately by your `curl` command (with `# Plan:` on Turn 1 and `# Thought:` on every turn). Everything needed is in `obs.json` and the calibrated reticle/grid drawn on `wrist_cam.png`.

## Visual Coordinate Reference (Annotated on wrist_cam.png)
- `+X` (UP in image): Away from robot base | `-X` (DOWN): Towards robot base
- `+Y` (LEFT in image): World left         | `-Y` (RIGHT): World right
- `+Z`: Height above table                | `Yellow numbers`: World (X, Y) in centimetres (2 cm grid at `grid_z`)
- `Cyan + crosshair`: Current 3D pinch center
- `Cyan [ + ] brackets`: Current jaw opening and closing line
- `Green line`: Current object long-axis orientation (`yaw_deg`: 0° = vertical along +X, 90° = horizontal along +Y)
- `pitch_deg = 90°`: Straight down (-Z) top-down grasp/place; `pitch_deg < 60°`: Tilted coaxial approach along finger axis

## Closed-Loop Turn Workflow
1. Inspect current state & view `wrist_cam.png`:
   `curl -s http://127.0.0.1:8081/obs` (returns `ee_pos`, `holding`, `status`, `workspace`, `wrist_cam_path`).
2. Select VLM call granularity upfront on Turn 1:
   - Coarse (2–4 calls): `look_at` -> `grasp` -> `place` -> `done`
   - Pre-Grasp Hover Verify (4–7 calls): `look_at` -> `goto(z=0.04)` (verify cyan `[ + ]` brackets straddle target) -> `grasp` -> `place_until_contact` -> `done`
   - Fine Visual Servoing (8–20 calls): `look_at` -> `grasp` -> pre-pose `goto` -> closed-loop `move_by` / `move_until_contact` -> `done`
3. Send motion primitives (`POST /plan`) and verify visually after every turn in the updated `wrist_cam.png`. If `status == "grasp_failed"`, call `robot.look_at(...)` to re-observe and recover closed-loop.

## Code-as-Policies Primitives (`POST http://127.0.0.1:8081/plan`)
robot.look_at(x=0.35, y=0.13, z=0.14, grid_z=0.015)
x, y = robot.deproject(u=0.52, v=0.48, z_plane=0.015)
robot.grasp(x=x, y=y, z=0.015, yaw_deg=-35, pitch_deg=90)
robot.place(x=0.24, y=0.06, z=0.075)
robot.place_until_contact(x=0.24, y=0.06, z=0.015, max_lag=0.012, relief=0.002)
robot.goto(x=0.28, y=0.05, z=0.04, arc_z=0.14, pitch_deg=25, azimuth_deg=90, roll_deg=0, speed=0.10)
robot.move_by(forward=0.025, dz=-0.002, dpitch_deg=-3, droll_deg=5, speed=0.04)
robot.move_until_contact(dz=-0.06, speed=0.03, max_lag=0.012, relief=0.002)
robot.open_gripper() / robot.close_gripper() / robot.home() / robot.done()

3. Omni 控制

Argon 在 Gemini Omni 世界模型中引导机器人，把散落的五块 Scrabble 字母牌（G, R, A, N, O）重新排成整齐的一行，拼出 A-R-G-O-N。Argon 不是给一条静态提示词，而是以闭环视觉回路进行引导：它撰写带时间戳的动作提示词，检查 rollout 的中间关键帧，并不断修正自己的提示词。

提示词：

Goal: {task_goal}

You are steering a video world model (Gemini Omni) in an iterative visual feedback loop. Inspect the attached frames from the latest rollout and write the next timestamped action prompt for the video world model:
- Break complex motions into one simple, single-direction action at a time.
- Keep inactive hands retracted and avoid blocking the camera's view of objects in the workspace.
- Explicitly state which areas of the workspace become empty after an object moves.
- Use [MM:SS.mmm] timestamps for each action beat and wrap your output in <omni_prompt> tags.

Example format (sample timestamps only, not the full prompt):
<omni_prompt>
[00:00.000] Character A's left hand rests at the side while the right hand reaches toward the target object.
[00:02.500] Character A's right fingertips push the object smoothly into the center of the table, leaving its starting area empty.
[00:05.000] Character A's right hand opens and retracts to the side so all placed objects remain in clear view of the camera.
</omni_prompt>

用 Argon 解锁 Real2Sim

1. 从机器人视角到仿真

Argon 观看一段真实机器人遥操作 episode，并构建数字孪生：它获取或创建在外观与接触行为上都与真实物体一致的仿真资产，供灵巧手操作。Argon 在一个迭代的视觉推理回路中调整资产与场景，使仿真的画面与接触事件和真实录制数据对齐。

提示词：

[Attached: robot_rollout_episode]

Build a MuJoCo scene of the robot and reconstructed task objects. When each episode is replayed frame by frame, by driving the robot joints toward the logged observation.state with joint-space PD.

Iterate using two verification objectives:
(1) Visual match: MuJoCo renders from the head_left, left_wrist, and right_wrist cameras line up with, and look like, those real observations.
(2) Physical match: the simulated contact force/torque on each Sharpa fingertip, in the sensor frame, matches the logged tactile_force in timing, magnitude and direction.

What you can tune happens before simulation starts:
(1) Static object assets: realistic geometry, collision shapes, mass, inertia, friction and joints, with appearance close to the real objects, plus the initial object poses at t = 0.
(2) Offline-planned robot commands, changed only in frames where the real fingertips are near an object. Each command must stay within 0.55 rad (arm joints) and 0.65 rad (hand joints) of the logged teleop trajectory. Everywhere else, replay the logged trajectory unchanged.

No sim cheating: task objects may move only through real MuJoCo contact with the robot hands and the environment (mj_step).
Not allowed after t = 0: changing MjModel parameters; applying external forces or torques to objects; welds or equality constraints; feedforward torques beyond joint-space PD; setting or rolling back object qpos/qvel; disabling or excluding collisions; reporting contact forces that don't come from the MuJoCo solver (for example, copying the logged tactile readings).

Immediately present the reconstructed scene and rollout results as an interactive HTML session viewer with one page per episode, showing:
- At the bottom: A synced timeline (play, scrub, speed control) that drives every panel.
- On the left panel: An interactive 3D Blender scene showing the simulated robot and objects, showing contact points and force arrows as replay progresses.
- On the right panel: Real vs. sim camera views side by side for all 3 cameras. Real vs. sim finger force visualizations.

2. 从人类视角到仿真

给定一段操作任务的第一人称人类视频，Argon 识别与任务相关的物体，用复合凸分解网格替换粗糙的包围盒，以表达凹形容器的内部可放置性，并把人手轨迹重定向到 MuJoCo 接触物理中一台 76 自由度双臂人形机器人（每只手为 22 自由度五指）上。

提示词：

[Attached: egocentric_human_video.mp4, candidate_sim_views.png, hand_3d_keypoints.json]

You are building a physically solvable 500 Hz MuJoCo manipulation environment (`scene.xml`) and studio Blender render that faithfully reproduces the tabletop task shown in `egocentric_human_video.mp4` on a bimanual humanoid robot with 22-DoF dexterous five-fingered hands:

1. Scene Reconstruction & Camera Calibration:
   - Identify only the manipulation-relevant objects (the grasped target object and the receiving container or landmark) and triangulate their 3D tabletop coordinates from the egocentric camera intrinsics/extrinsics.
   - Replace crude bounding boxes with compound convex-decomposed 3D meshes so concave containers (nested bowls, recessed wooden trays, open bins) support physical interior placement.

2. 500 Hz Dexterous Motion-Fitting Stress Test:
   - Retarget the 3D human wrist and finger keypoints to the humanoid's 7-DoF arm and 22-DoF hand joints, and step the trajectory in MuJoCo at 500 Hz (dt = 0.002s) with zero kinematic weld constraints.
   - Verify opposing thumb-and-finger force closure, clean object lift (> 2 cm), and stable final placement inside or beside the target landmark.

3. 从草图到仿真

借助 Argon，开发者可以拍一张粗糙的纸笔草图——画着一台 Apollo 2 人形机器人沿抛物线掷球、推倒桌上 3-2-1 堆叠的箱子——Argon 会自动构建 GPU 加速的 mujoco_warp 场景，合成平衡的 76 自由度全身与五指投掷轨迹，在闭环物理中验证 6/6 全部击倒目标，并导出 Blender 4.5 Cycles 渲染。

提示词：

[Attached: hand_drawn_throw_sketch.jpg]

Translate this hand-drawn sketch into a physically verified humanoid simulation in `mujoco_warp` and a raytraced Blender Cycles video:

1. Scene & Kinematic Setup from Sketch:
   - Parse the spatial layout from `hand_drawn_throw_sketch.jpg`: place the 76-DoF humanoid (32-DoF body equipped with two 22-DoF five-finger dexterous hands) at the origin facing a table holding a 3-2-1 pyramid of 6 rigid target boxes, with a projectile sphere grasped in the right five-finger hand.

2. Whole-Body Balance & Dexterous Throw Synthesis:
   - Synthesize a 50 Hz whole-body throwing trajectory (140 steps, 2.80 s) across four phases: (1) stable stance & 5-finger power grasp closure, (2) torso/shoulder wind-up with ankle-hip Center-of-Mass compensation, (3) forward whipping acceleration, and (4) synchronized 22-DoF five-finger opening at peak hand velocity so the ball launches on a ballistic arc into the pyramid's second-tier center of mass.
   - Verify in `mujoco_warp` contact physics that the humanoid maintains upright balance (< 3 cm pelvis drift) and knocks down all 6 target boxes.