Democratizing Robot-Use Agents: A Unified Harness with Open Models

Preliminaries: the robot-use contract

A robot-use agent receives an event stream rather than a timeless image. Each observation includes RGB, camera calibration, robot state, robot description, and task instruction. Each action declares a target pose, gripper opening, execution budget, and resource ownership. The environment service assigns a state version, and every action is checked against the version on which it was planned.

A tool is more than a thin API wrapper. It may translate one model intent into several low-level environment steps and monitor pose error, planning rejection, unsafe displacement, divergence, and stalls along the way. Its result must distinguish immediate completion, queued work, running work, failure, cancellation, and uncertain execution. These states require different recovery choices and cannot be compressed into a generic error message.

RoboDojo contains 42 simulated tasks covering generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following. The main evaluation uses three seeds and two layouts per task, for 252 episodes. Agents receive RGB, calibration, proprioception, robot descriptions, and instructions, but not simulator ground-truth depth, CAD assets, hidden annotations, task-verifier feedback, or web access.

Method

A five-layer boundary between models and environments

The architecture separates the agent, orchestration, tool, service, and backend layers. The agent layer manages model interaction and context. Orchestration owns resource scheduling and job lifecycle. Tools expose stable robot semantics across environments. Services manage episode, perception, and action-model resources. Backends implement concrete simulators, real robots, and model adapters. Requests move downward through explicit interfaces and results return along the same path.

The benefit is a small replacement boundary. A new model does not require a new robot API. A new simulator replaces its service and backend adapter without rewriting the agent prompt. A new gripper implementation can preserve the same visible action contract. The paper describes this as shared tool semantics, not a requirement that every environment share the same low-level controller.

flowchart TB
    A[Agent Runtime
model and context] -->|tool request| O[Orchestration
jobs and resource queues]
    O --> T[Shared Tools
observe move grasp release]
    T --> S[Services
episode and perception]
    S --> B1[Simulator Backend]
    S --> B2[Real Robot Backend]
    S --> B3[Action Model Backend]
    B1 -->|state images receipts| S
    B2 -->|state images receipts| S
    B3 -->|prediction and diagnostics| S
    S -->|typed result| O
    O -->|job handle or completed output| A
    C[Context Policy
sliding image window] --> A
  

Shared tools avoid a combinatorial maintenance problem

The paper defines tools by stable semantics such as observing the scene, moving an arm, controlling a gripper, releasing an object, and ending an episode. Environment adapters map those meanings to native interfaces. Without this split, every tool-environment pair can require its own implementation, creating an N times M maintenance surface. Shared semantics reduce the repeated work to the tool contract and the environment adapter, so adding a model or device does not duplicate action logic.

Tool outputs carry images, success flags, errors, job handles, state versions, stop reasons, and diagnostics rather than only free-form text. A common result shape allows the same model prompt to consume simulator and real-robot evidence and allows orchestration to recognize work that remains active.

Figure 3: Shared tool semantics and environment adapters avoid separate implementations of every tool in every backend.

Asynchronous scheduling and a two-stage pipeline

Waiting for every action wastes the time in which the model could prepare the next decision. Never waiting is equally unsafe because consecutive commands can contend for the same arm. The harness separates return mode from execution placement. An immediate call waits for completion; a background call returns a job handle. Exclusive calls for one resource enter a FIFO queue, while independent resources may execute in parallel.

This creates two overlapping stages. The model can reason while a submitted physical action runs, and sequential actions remain ordered behind the same resource. In five real-world tasks, pipelining reduced episode duration by 4.8% to 73.7% compared with serial execution. The fraction of physical-action time overlapping a model request ranged from 84.5% to 93.8%, whereas serial episodes had no overlap.

Figure 4: Serial execution waits after each physical action. The pipeline overlaps model requests with action execution while resource queues preserve the order of commands that share a robot.

Immutable observations keep pixels, calibration, and robot state aligned

After an arm moves, a pixel in an earlier image still belongs to the earlier camera pose. Interpreting that pixel with a newer calibration or robot state mixes two physical moments. The harness stores each observation as an immutable snapshot containing images, intrinsics and extrinsics, robot state, environment version, and observation sequence. Localization and motion tools operate from that recorded coordinate relationship.

This matters because grasp points flow through several transformations. A pixel becomes a ray, the ray is intersected with geometry or another view, the point is transformed into the robot base, and the result becomes a target pose. A stale image-to-calibration relationship contaminates every later step. Immutable snapshots make that invalid combination unrepresentable at the interface.

Request receipts distinguish no execution from unknown execution

Physical actions are not idempotent. If a reply is lost, the command may never have started, or it may already have changed the world. Retrying blindly can move the arm twice. The harness records request identity, episode sequence, execution status, and state version. Duplicate requests are rejected or resolved to the existing result.

When communication is interrupted and completion cannot be determined, new physical actions are blocked while the session asks the service to reconcile the request. If reconciliation fails, the result remains explicitly uncertain. Treating uncertainty as a first-class state prevents a retry from turning a transport error into a second physical action.

Joint motion and gripper control

Separate move and close calls force the model to coordinate two asynchronous jobs. Closing before contact can produce an empty grasp; moving before a close finishes can leave the object unsecured. The paper's move tool accepts a target pose and an optional gripper opening in one request, allowing the arm and fingers to change together within a bounded action.

The visible tool remains a single semantic operation even though the service receives multiple primitive steps. This reduces tool-call overhead and prevents a downstream action from starting before the intended grasp state has been requested.

Monitored motion with explicit stopping criteria

The motion tool does not rely on a fixed step count. After every primitive step it measures pose error and limits angular motion using the rotation radius. For maximum translation step delta and rotation radius r, the angular bound is:

$$\theta_{\max}=\begin{cases}\min\{0.6,\,2\arcsin(\min\{1,\delta/(4r)\})\},&r>0,\\0.6,&r=0.\end{cases}$$

Rotation consumes part of the same displacement budget, leaving the following translation allowance:

$$\delta_p=\max\{0,\delta-2r\sin(\Delta\theta/2)\}.$$

This prevents a large orientation change from sweeping the end effector through a displacement that was intended for only the next translation waypoint. The tool selects a bounded rotation first and spends the remaining budget on translation toward the target.

Orientation error uses the shortest quaternion distance. For unit quaternions q1 and q2, the paper measures:

$$d_q(q_1,q_2)=2\arccos(\min\{1,|q_1^\top q_2|\}).$$

The absolute inner product is necessary because q and negative q represent the same orientation. Translation and rotation errors are combined into a progress score:

$$v_t=e_t^p+0.1e_t^q,$$

A run is marked as diverging after repeated deterioration:

$$v_t>v_{t-1}+0.01.$$

Stall detection compares the improvement over the last four measurements. Motion stops when neither translation nor orientation is making meaningful progress:

$$e_{t-3}^p-\min_{j=t-2}^{t}e_j^p\leq0.1\epsilon_p,\qquad e_{t-3}^q-\min_{j=t-2}^{t}e_j^q\leq0.1\epsilon_q.$$

The reported stop reason is one of pose reached, planning rejected, diverging, stalled, unsafe motion, or native end. The model therefore receives information about why control ended, not merely a generic failure. It can lower the grasp, choose another target, or abandon the current plan.

Numerical and visual feedback answer different questions

Reaching a target pose does not prove that an object was grasped or placed. The fingers may close at the correct coordinates without contact, or a placement may push the object away. The harness returns pose errors, gripper state, stop reason, and post-action images. The numbers describe whether control completed; the images describe whether the object changed as intended.

Images alone make millimeter residuals difficult to judge. Numerical state alone hides slipping, collisions, and poor grasp geometry. Binding both to the same snapshot lets the model separate "the controller executed" from "the task progressed."

A sliding image window preserves useful text history

Long episodes accumulate images whose relevance decays, while instructions, calls, and action results remain useful. The sliding-window policy removes image nodes only from older tool outputs and replaces them with text placeholders containing the tool name and call identifier. User images, calls, textual outputs, and surrounding history remain intact.

Images are removed in fixed batches rather than one at a time. The shared prompt prefix therefore stays stable between eviction rounds, supporting cache reuse. If image eviction is insufficient, the policy removes complete interaction blocks and never separates a tool call from its result. It does so only after the image window has already reduced the context.

Figure 5: Six design choices address physical consistency, action precision, and context management: immutable observations, request receipts, joint motion and gripper control, monitored movement, multimodal feedback, and a sliding image window.

Repository correspondence

The public repository implements the resource boundary in uni_robot_agent/execution/scheduler.py. ResourceScheduler.submit() sends parallel work to a bounded pool and appends exclusive work to a per-resource FIFO queue. flush() cancels only jobs that are still queued; it does not stop a physical action that has already started. That behavior matches the paper's separation between cancellation and uncertain execution.

# uni_robot_agent/execution/scheduler.py
def flush(self, resource_key: str) -> int:
    with self._lock:
        queue = self._queues.get(resource_key)
        cancelled = 0
        for job_id, _ in list(queue):
            cancelled += int(self._registry.cancel_if_queued(job_id))
        queue.clear()
    return cancelled

The waypoint arithmetic appears in uni_robot_agent/tools/gripper/pose.py, while motion.py contains the tracker that evaluates arrival, divergence, and stalls. The implementation uses the absolute quaternion dot product and the same 0.1 orientation weight described in the paper.

# uni_robot_agent/tools/gripper/pose.py
def quaternion_distance(left, right):
    dot = float(np.clip(abs(np.dot(quaternion(left), quaternion(right))), 0, 1))
    return 2 * math.acos(dot)

# uni_robot_agent/tools/gripper/motion.py
score = position_error + .1 * orientation_error
self.worsening = self.worsening + 1 if score > self.prior_score + .01 else 0
if self.worsening >= 2:
    return "diverging", { ... }

The observation hub in observation/hub.py stores frozen snapshots with sequence and world version, and context/policies/sliding_window.py implements batched image eviction with text placeholders. The mapping is close enough to identify the intended architecture, but this reading did not execute the experiments or claim a reproduction of the reported numbers.

Results

RoboDojo: an open model beats the published closed-model baseline

Across 42 tasks and 252 main episodes, Qwen3.8-Flash-Next with the harness reaches 25.00% success rate and 31.39 score. It surpasses the published GPT-6 Astra with RoboProbe result of 22.48% and 28.97, as well as several VLA and world-action baselines listed by the authors. The key point is not only the 2.42-point score margin. An open-weight model enters a competitive range in a zero-shot setting without privileged inputs or distillation from the closed model.

The stronger GPT-6 Astra reaches 46.67% and 51.90 through the same harness, showing that the design is not beneficial to only one open model. Qwen reaches just 1.90 with RoboProbe and 7.75 with RPent, while GAP Hybrid reaches 21.01. Moving the same model into this harness produces 31.39. That contrast points to the observation, execution, and feedback contract rather than language-model choice alone.

Configuration	Success rate	Score	Role
GPT-6 Astra + RoboProbe	22.48%	28.97	Published closed-model baseline
Qwen3.8-Flash-Next + GAP Hybrid	16.11%	21.01	Another harness
Qwen3.8-Flash-Next + uni-robot-agent	25.00%	31.39	Main result without depth
DeepSeek-V4.1-Flash + uni-robot-agent	21.39%	27.11	Second open-weight model
GPT-6 Astra + uni-robot-agent	46.67%	51.90	Closed model in the same harness
GPT-6 Astra + harness + depth	52.36%	56.76	Reference depth ablation

The dimension breakdown is uneven. Qwen is stronger than many end-to-end policies on memory and open-vocabulary tasks, but the authors report that it remains below Xiaomi-Robotics-1 and OpenWAM-alpha on long-horizon execution. The harness turns semantic understanding into useful behavior, yet it does not remove state tracking, contact, or multi-stage error accumulation. Astra reaches 31.94% generalization success versus 30.50% with RoboProbe, and its harness configuration improves every listed RoboDojo dimension over Astra with RoboProbe, with especially large gains in memory and precision.

Nineteen real-world tasks: the open model approaches a fine-tuned policy

The real-world study contains 19 tabletop manipulation tasks grouped into spatial understanding, object generalization, instruction following, compositional generalization, and unseen tasks, with 20 points per dimension. Qwen with the harness scores 49.50, close to the 47.75 of pi-0.5 fine-tuned for 10 hours of demonstrations per task, and above the zero-shot 35.50 of Dream-1. GPT-6 Astra with the same harness reaches 75.50.

The comparison is not sensor-matched. The harness agents additionally receive depth from D405 cameras, whereas pi-0.5 and Dream-1 consume RGB. The 49.50 versus 47.75 difference cannot be assigned cleanly to model or harness. The task profile remains informative: the agent scores 13.00 on compositional generalization versus 6.00 for pi-0.5 and 10.00 for Dream-1, but only 6.00 on object generalization versus 12.50 for pi-0.5.

Method	Spatial	Object	Instruction	Composition	Unseen	Total
pi-0.5, task fine-tuned	11.25	12.50	14.00	6.00	4.00	47.75
Dream-1, zero-shot	4.00	8.00	9.50	10.00	4.00	35.50
GPT-6 Astra + harness	17.00	12.50	18.00	20.00	8.00	75.50
Qwen3.8-Flash-Next + harness	9.00	6.00	13.50	13.00	8.00	49.50

Real Eval awards partial credit for completed subtasks. Its total therefore measures graded progress rather than only final success. The near match between 49.50 and 47.75 should not be read as identical one-shot success. It indicates that the agent route can complete substantial portions of compositional and unseen tasks, while object-level generalization remains a clear weakness.

Depth and the skill library are not the main source of performance

Depth is tested at three levels: RGB with depth and calibration, RGB with calibration, and RGB only. Astra moves from 55.83 to 55.04 without depth, retaining 98.6%. Qwen moves from 35.89 to 31.39, retaining 87.5%. With calibration but no depth, the model can still triangulate corresponding pixels across views. Without calibration, Astra falls to 31.29 and Qwen to 9.13. Camera parameters are the more important spatial bridge in this setup.

Removing the robot skill library has a similarly small aggregate effect. Qwen retains 96.6% of its harness score, reaching 30.31 and 24.58% success when skill access and skill writing are disabled. Task instructions, environment descriptions, and standard robot tools remain. The ablation therefore tests whether reusable skills encode task-specific solutions; it does not remove the basic observation and motion interface.

Ablation	Astra score	Qwen score	Interpretation
RGB + depth + calibration	55.83	35.89	Depth reference upper bound
RGB + calibration	55.04 (98.6%)	31.39 (87.5%)	Multi-view triangulation replaces simulator depth
RGB only	31.29	9.13	Spatial grounding collapses without camera parameters
Qwen without skill library	-	30.31 (96.6%)	Standard tools remain and performance is nearly retained

Cost, tokens, and wall-clock time

Qwen costs an estimated $1.09 per episode, compared with $21.29 for Astra, a roughly 20-fold difference. The resource paths differ as well. Reaching a score of 25 takes Astra about 7.4 million tokens and 42.6 minutes, Qwen about 23.5 million tokens and 95.5 minutes, and DeepSeek about 49.9 million tokens and 99.2 minutes. Open weights reduce per-token cost, but they do not automatically reduce inference time or total token count.

Pipelining improves wall-clock time without reducing the amount of model computation. In five Astra tasks, each tested once under the same seed and layout, every pipelined episode is shorter than its serial counterpart. swap_T falls from 34.7 minutes to 12.5, and press_by_number from 25.0 to 13.7. The result demonstrates the mechanism, but one episode per condition is not enough for a stable average speedup estimate.

Model capabilities and failure modes

Figure 6: RoboDojo performance across generations, families, and parameter scales. Robot use is not limited to the newest or largest model in every family; training distribution and generation effects are substantial.

Robot use does not appear only in the latest model generation. GPT-5.6 Sol already reaches 19.17% and 24.04, while GPT-6.1 Sol reaches 37.50% and 45.63. Qwen3.5 through 3.7 variants record zero success, but Qwen3.8-27B reaches 8.06% and Qwen3.8-Flash-Next reaches 25.00%. Parameter count alone does not explain the ordering; the authors argue that training distribution and capabilities within a generation matter more.

Trajectory analysis identifies four failure groups: general visual understanding, fine-grained spatial correspondence, geometry code, and embodied reasoning. Early Qwen models misinterpret objects or outcomes in 12.2% to 23.8% of trials. Even GPT-6.1 Sol, with the lowest spatial failure rate among the evaluated models, has 31.5% of corresponding-pixel selections rejected. GLM-5.3 Flash proposes physically inappropriate grasp poses in 26.2% of analyzed pick-and-place trials.

The paper correlates RoboDojo score with MMMU-Pro, ERQA, ScreenSpot-Pro, and LiveCodeBench v6. All four associations are positive. Embodied reasoning and general visual understanding are relatively strong, while 2D screen grounding is a weaker predictor of 3D robot grounding. The analysis contains six models and is best treated as hypothesis-generating rather than a calibrated predictor.

Limitations

The authors explicitly state that precise manipulation and reliable long-horizon execution remain difficult. The harness improves observability and recovery, but contact-rich control, millimeter-level placement, and long sequences still accumulate errors. Qwen's weaker long-horizon result compared with some world-action models supports that concern.

Real Eval uses a rubric that grants partial credit rather than measuring only final task success. The present score is useful for comparing graded progress, but it cannot be treated as an equivalent success-rate comparison. Per-task criteria and capability subtotals should be read alongside the total.

The real-world methods do not receive identical inputs. Harness agents use depth, while the two baselines use RGB. The paper discloses this difference, but it prevents a clean decomposition of the performance gap into model, sensor, and harness contributions. A sensor-matched comparison would isolate those factors more convincingly.

Several cross-model evaluations use one seed and one layout. The main Qwen, DeepSeek, and Astra entries have larger repetition, but generation and family comparisons are mostly single-setting, and the Astra depth ablation is limited by evaluation cost. Individual model rankings may be sensitive to layout or seed.

The capability correlation and embodied-reasoning analysis are narrow. The correlation uses six models, and embodied reasoning is approximated mainly by whether a proposed grasp pose is plausible under GPT-6 Astra evaluation. Bimanual coordination, contact ordering, tool selection, and other forms of physical reasoning are not captured by that metric.

This review inspected the paper and the public code at one commit, but did not run a full experimental reproduction. The source files support the described architecture and motion arithmetic; they do not independently validate every reported benchmark number.

Conclusion

The central contribution is not another end-to-end action network. It is a set of explicit system boundaries for robot use: the model decomposes and decides, tools turn intent into monitored physical actions, services own state and resources, and backends implement concrete devices. Shared semantics, asynchronous scheduling, immutable observations, request receipts, and structured feedback make an open model viable in that loop.

The results show that open-weight models already possess a useful basis for robot manipulation when calibration, robot descriptions, and reliable feedback are available. They also show why genuine physical capability is still incomplete: visual correspondence, geometric computation, bimanual coordination, and memory remain bottlenecks. A promising next step is to use the harness to collect robot interaction data with failure reasons and multimodal receipts, then train models on the capabilities that execution trajectories expose.

An open model does not need to become a robot policy first; it needs an execution environment that turns general reasoning into observable and correctable physical action.