Democratizing Robot-Use Agents: A Unified Harness with Open Models

Democratizing Robot-Use Agents with a Unified Harness

Source: Democratizing Robot-Use Agents: A Unified Harness with Open Models, OpenRUA Team, November 3, 2026.

Robot manipulation is developing a path that differs from training a single end-to-end vision-language-action policy. Instead, a general-purpose model reads a task, calls tools, observes the results, and decides what to do next. Systems of this kind are often called robot-use agents. They borrow the planning, decomposition, and recovery abilities of frontier models and turn language and visual understanding into sustained physical action.

The problem is that the strongest models are usually available only through proprietary APIs. Researchers cannot access their weights or continue training them directly on robot experience. Large-scale interaction is expensive, and model-level optimizations such as efficient caching remain under the provider's control. Open-weight models offer an alternative, but they often trail on spatial understanding and long-horizon agentic behavior. The project team argues that before attributing failures to the model, the field must answer a more basic question: if a unified, reliable harness avoids privileged inputs and still works well, how capable are open models at robot use today?

OpenRUA's uni-robot-agent is built around that question. It places models, robots, tasks, and simulated environments behind one shared tool interface, while preserving the link between observations, calls, and feedback for future training. The central conclusion is direct: well-structured information, reliable tools, and informative execution feedback can substantially improve how general models use robots. On RoboDojo, Qwen3.8-Flash-Next reaches 31.39 Score and 25.00% success rate, exceeding the published Astra-RoboProbe result of 28.97. Across 19 physical dual-arm tabletop tasks, Qwen reaches 53.50 zero-shot and Astra reaches 80.50.

These results show that open models can be organized into effective robot-use agents. They do not show that general-purpose robotics has been solved. The physical evaluation is limited to tabletop manipulation with two arms, and OpenRUA additionally uses D405 depth in this comparison while the two baselines use RGB only. More importantly, the README states that the public package does not yet include a supported physical-robot connection workflow. ARX One and Franka integrations are marked as coming soon. The paper contains real-robot experiments, but readers cannot yet reproduce that hardware deployment by following the public installation instructions.

The central result: the harness changes what a model can do

The most important comparison is not any single score. It is the change produced by replacing the harness around the same model. The project page reports that Qwen3.8-Flash-Next rises from 1.90 Score under RoboProbe to 31.39 under uni-robot-agent, a gain of 29.49 points, without depth in the main setting. With Astra, the same harness reaches 51.90 on RoboDojo. Model capability still matters, but a system that produces executable actions, stable evidence, and accurate context is itself part of the capability.

RoboDojo configuration	Score	Success rate	Notes
Qwen3.8-Flash-Next + GAP Hybrid	21.01	16.11%	The strongest default Qwen baseline; GAP incorporates pi-0.5.
Qwen3.8-Flash-Next + uni-robot-agent	31.39	25.00%	Main setting: RGB, calibration, proprioception, and robot descriptions, with no depth.
DeepSeek-V4.1-Flash + uni-robot-agent	27.11	21.39%	Shows that the result is not confined to one open-weight model family.
GPT-6 Astra + uni-robot-agent	51.90	46.67%	Main result over the same 252 episodes.
Astra-RoboProbe	28.97	Not reported in the same main table	A separate published result; its original protocol is retained.

The final row requires care. Astra-RoboProbe is a published reference under a different harness, not a controlled rerun produced by this project. Every baseline retains its own protocol, so the rows are useful for mapping the capability landscape but are not a strict causal comparison. Even so, the large within-model change for Qwen demonstrates that asking only whether the model is strong enough omits a major part of the system.

A five-layer architecture with replaceable boundaries

A tool call passes through five layers. The agent layer manages model interaction, message history, and context. Orchestration schedules requests, controls concurrency, and dispatches tools. The shared tool layer exposes motion, perception, skill, and episode operations. Services manage simulator, camera, and action-model resources at runtime. Concrete simulators, robots, and model implementations sit in the backend layer.

These layers are more than software organization. They define what a researcher must change when testing a new idea. Switching models should not force a rewrite of robot control. Adding an environment should require an adapter, not a separate implementation of every tool. Replacing a tool or adding an action model should leave the high-level interaction record comparable. For a system intended to support cross-environment training, those boundaries matter more than a single benchmark number.

A shared interface represents both actions and evidence

The common tools do not assume that every robot is the same. Environment descriptions identify the available arms, cameras, and optional capabilities. Motion tools accept a target pose, coordinate units, and an optional gripper opening. Perception tools return images, calibration, or measurements. Episode tools save trajectories and determine when a task ends. Adapters translate shared operations into native simulator or robot commands.

The same semantics can express tasks across different arm configurations and objects. "Stack the red block on the blue block" can be decomposed into locating both blocks, grasping and aligning, releasing, and checking the result. A bimanual handover can be expressed as presenting the object, establishing a second grasp, transferring it, and releasing. Tool names, parameter units, and result shapes remain consistent, so the model does not have to learn an unrelated protocol for every environment.

This also connects the harness to agentic training. Systems such as AgentGym collect trajectories across environments in a common interaction format and use filtered demonstrations for supervised fine-tuning. Agent Lightning separates structured invocation records from execution and extracts inputs, outputs, and rewards for policy optimization. The present contribution is not a new training algorithm. It first binds observations, tool calls, and feedback into a consistent record and makes experience from different environments easier to combine. Data selection, learning objectives, and reward design remain open questions, and the paper explicitly leaves training on this experience to future work.

Reasoning and execution form a two-stage pipeline

Robot agents often waste time waiting for every movement to finish. uni-robot-agent distinguishes immediate calls from background calls. An immediate call waits for completion. A background call returns a job handle as soon as work begins, allowing the agent to reason while the tool runs. Calls that control the same arm enter a queue and execute in submission order, preventing two movements from fighting over one resource. Independent operations, such as reading an existing snapshot, can overlap with motion.

If new feedback changes the plan, flush_queue cancels queued calls for the specified resource that have not started, while leaving active work in place. The agent can then submit corrected actions. The paper reports 4.8% to 73.7% shorter runs with pipelining on five selected tasks. Each configuration contains one episode, and API latency, resource sharing, and different action sequences all affect duration. The full reduction therefore cannot be attributed to scheduling alone.

What remains possible without privileged inputs

Robot benchmarks can accidentally give an agent information that a physical robot would never have, including simulator object coordinates, hidden scene annotations, task-verifier output, or web search. The main uni-robot-agent setting permits only RGB images, camera calibration, robot proprioception, robot descriptions, and task instructions. It excludes depth, object CAD assets, hidden annotations, verifier feedback, and external web access.

The restriction also applies to tools. Localization must derive coordinates from permitted observations. The agent can move the camera, find corresponding points in multiple views, and triangulate using calibration. Robot poses and execution outcomes can return as measurements, but success checks from the task verifier remain outside the agent and are used only for evaluation.

The team manually checks for access to prohibited information and adds small clarifications to some RoboDojo tasks. Certain instructions omit or ambiguously describe requirements, such as how many times each red button must be pressed or how blocks and boards must be arranged in a tower. The additions state task rules without revealing scoring rubrics or verifier logic. This is a useful reminder that zero-shot does not mean an agent can infer hidden benchmark conventions. The task specification must be clear; otherwise the evaluation measures riddle comprehension.

Six mechanisms keep the next decision attached to physical state

A physical interaction can leave the agent with an outdated image, an uncertain action status, or incomplete evidence of success. The harness addresses these risks with six design choices.

1. Immutable snapshots. Images, calibration, and robot state are bound to the same snapshot. If the robot moves before an old image is localized, mixing the old pixels with a new camera pose produces systematic coordinate errors.
2. Request identity and deduplication. A missing reply does not prove that an action never ran. Every request has an identity, and execution records reject duplicates. When completion is uncertain, later actions are blocked while the system recovers the record; unresolved requests remain explicitly uncertain.
3. Combined motion and gripper control. Separate commands cannot adjust the gripper during a movement. The motion tool accepts a target pose and optional gripper opening together, allowing the arm and fingers to act at the same time.
4. Measured, iterative motion. A fixed step count may stop too early or waste effort on an unreachable target. The tool generates intermediate targets from measured poses and checks after each step for arrival, stalling, or an execution limit.
5. Outcome interpretation. Reaching a pose does not prove that an object was grasped. Pose error, stop reason, and outcome images together reveal what happened, so the agent can refine the motion or change strategy.
6. Context compaction. Older images become text placeholders in batches, preserving recent views and a stable prompt prefix for cache reuse. If more space is needed, older complete interactions are removed while calls remain paired with results and essential instructions and pending calls are protected.

These may look like implementation details, but they determine whether an agent can recover. A model may still have a chance after a failed grasp. If the system loses action results, permits duplicate execution, or lets old images participate in localization, that failure can become an undiagnosable state error.

RoboDojo: capability profiles of open models

Simulation evaluation uses 42 RoboDojo tasks spanning generalization, precision, long-horizon execution, memory, and open-vocabulary instruction following. The main configuration uses three seeds and two layouts per seed, giving 252 episodes per configuration. Score credits partial progress, while success rate counts only fully successful episodes. Each episode has a three-hour wall-clock limit because aggregate performance in four-hour runs largely plateaus by the third hour.

Qwen3.8-Flash-Next with uni-robot-agent reaches 31.39 Score and 25.00% success rate, versus 21.01 and 16.11% for the strongest Qwen baseline, GAP Hybrid. DeepSeek-V4.1-Flash reaches 27.11 and 21.39%, showing that the result extends beyond one Qwen model. Astra reaches 51.90 and 46.67%, confirming that the harness also supports much stronger frontier models.

Different models have different capability profiles. Qwen and DeepSeek perform relatively well on memory and open-vocabulary tasks but trail on precision and long-horizon execution. Xiaomi Robotics-1 and OpenWAM-alpha reach 38.39 and 34.93 on long-horizon tasks, but only 7.81 and 10.41 on memory. Current foundation models are not balanced across all five dimensions. Precise manipulation and long-horizon planning remain central weaknesses.

Small samples change rankings, so variance belongs in the report

The paper also studies a separate depth-enabled Qwen pool with 50 rollouts for each of 54 task variants, totaling 2,700 episodes. The full-pool Score is 34.59. With six random rollouts per variant, the mean sampled Score is 34.59 and the central 95% range is 30.65 to 38.51. Taking the first six recorded rollouts ordered by pass and layout identifiers instead produces 36.55, an upward difference of 1.96 points.

This does not make 30.65 to 38.51 a confidence interval for the main 31.39 result. The pools and settings differ. The check shows that a small subset can land near the aggregate mean on average while individual ordered subsets vary by several points. Robot-agent evaluation is expensive, so many studies must use few episodes. Reporting sample size, sampling rules, and variance alongside the best success rate makes the evidence substantially more credible.

Physical robots: zero-shot operation across 19 tasks

The physical evaluation uses 19 entries on a dual-arm tabletop platform. Five dimensions contribute 20 points each: spatial understanding, object generalization, instruction following, compositional generalization, and unseen tasks. Task criteria award credit for completed objectives and, where specified, for intermediate progress such as grasping before placement.

OpenRUA evaluates Astra and Qwen zero-shot on the harness. Neither agent receives task-specific training on these entries. The baselines are pi-0.5 fine-tuned on demonstrations from the same platform and Dream-1 evaluated zero-shot through its public API. One input difference must remain visible: the OpenRUA agents use Intel RealSense D405 depth, while pi-0.5 and Dream-1 use RGB only. The result demonstrates that the system works on physical hardware, but it is not a completely matched model ranking.

Physical-robot configuration	Total score (100)	Task training	Inputs
GPT-6 Astra + uni-robot-agent	80.50	Zero-shot	RGB, depth, calibration, proprioception, and robot descriptions
Qwen3.8-Flash-Next + uni-robot-agent	53.50	Zero-shot	RGB, depth, calibration, proprioception, and robot descriptions
pi-0.5	47.75	Fine-tuned with 10 hours of robot data per task	RGB
Dream-1	43.50	Zero-shot	RGB

Astra scores 20/20 on compositional generalization, but both OpenRUA agents score only 8/20 on unseen tasks. General models can recombine known skills around new tabletop goals, yet still struggle when the structure of a task is genuinely out of distribution. The physical benchmark is also smaller than the simulation suite, so these numbers do not establish reliability in open homes, mobile manipulation, or long industrial workflows.

Four recurring bottlenecks

The paper audits trajectories from 13 models and groups failures into four recurring capabilities. These are not substitutes for benchmark scores. They identify where future robot-specific training may need to improve.

Visual understanding

Earlier Flash and Plus models misidentify objects or outcomes. They read a zero as nine, for example, or treat an inverted cup as upright. These failures affect 12.2% to 23.8% of reviewed trials. Qwen3.8-27B falls to 3.2%, while Qwen3.8-Flash and GPT-5.6 Sol show none in the reviewed trials. Visual interpretation is the entry point to the action loop; if the object state is wrong, later geometry can be precise and still aim at the wrong thing.

Cross-view spatial grounding

The agent must identify the same physical point across camera views before triangulating it. Inconsistent image points create large reprojection errors, causing the tool to reject the result. Even GPT-6.1 Sol, the best model in this comparison, fails 31.5% of its triangulation calls. Clicking a target on a two-dimensional screen is not the same capability as locating a point in three-dimensional space, which helps explain why a general grounding benchmark cannot fully predict manipulation performance.

Geometric code

Pose computation combines coordinate frames, rotation matrices, quaternion conventions, and unit conversion. Earlier Flash and Plus models reverse transforms or mix conventions. Confirmed errors affect 47.3% to 81.8% of their executed calculations. The rates fall to 8.2% for Qwen3.8-Flash, 1.5% for DeepSeek, 0.7% for GPT-5.6 Sol, and no observed errors for GPT-6.1 Sol. Syntactically valid code is not the same as geometry that respects robot conventions.

Embodied reasoning

A reachable pose can still be a poor grasp. The paper focuses on pick-and-place tasks and uses GPT-6 Astra to judge whether proposed grasps are physically reasonable. GLM leaves inappropriate grasp proposals uncorrected in 26.2% of reviewed tasks, repeatedly failing to reorient its wrist. Newer Qwen and GPT models make fewer such errors, but Astra still struggles with complex bimanual coordination. Embodied reasoning here is concrete: it must combine object shape, gripper geometry, contact, and the motion that follows.

These failures also correlate with general capabilities. Across six models and all 42 RoboDojo tasks, the Pearson correlation with robot-use Score is 0.97 for ERQA, 0.93 for MMMU-Pro, 0.69 for ScreenSpot-Pro, and 0.75 for LiveCodeBench v6. Correlation does not show that any one benchmark causes robot performance, but all four abilities participate in the same action loop: interpreting observations, identifying targets, expressing decisions as code, and reasoning about physical consequences.

How much depth and calibration help

A separate ablation compares RGB with depth and calibration, RGB with calibration only, and RGB alone. Proprioception remains available in every condition. With depth, the agent can map a pixel directly to a 3D position. With calibration alone, it can move the camera and triangulate, but must find corresponding points. Without either, it must infer spatial relations from images and use action feedback to correct mistakes.

This ablation is not a direct reproduction of every model's main result. Qwen uses 252 episodes across three seeds and two layouts. Astra uses one seed and one layout, or 42 episodes, to limit evaluation cost. The 55.83 in the table is therefore Astra's ablation setting, not the 51.90 obtained over 252 episodes in the main experiment.

Model and inputs	Qwen Score (252 episodes)	Astra Score (42 episodes)
RGB + depth + calibration	35.89	55.83
RGB + calibration, no depth	31.39	55.04
RGB only	9.13	31.29

Without depth, Astra retains 98.6% of its Score and Qwen retains 87.5%. Astra is better at recovering spatial information by moving the camera and comparing views. Removing calibration as well lowers the two models to 31.29 and 9.13. In the RGB-only condition, Astra completes 10 of the 42 tasks, a wider task range than Qwen achieves.

The appendix examines two successful Qwen episodes without depth or predefined skills. In one, the agent writes and executes geometry code to triangulate test-tube positions across views. In another, it tests candidate table heights against image evidence. These cases show that calibration and code tools can reconstruct RGB-based localization, but at the cost of additional reasoning steps and greater exposure to failure.

The skill library helps, but is not the source of the result

To test whether the system depends on handcrafted task procedures, the team removes Qwen's access to both the robot skill library and skill writing. Task instructions, environment descriptions, and standard tools remain. Without skills, Qwen reaches 30.31 Score and 24.58% success rate. With skills, it reaches 31.39 and 25.00%. The Score gap is only 3.4%, and the success-rate gap is 0.42 percentage points.

This does not mean skills are useless. It means predefined skills are not a prerequisite for the harness. Language-capable agents can interpret some requirements that VLA policies normally receive through demonstrations, though demonstrations remain better for contact details that are difficult to describe in prose. Benchmark design must distinguish an agent that reasons and acts from one that receives a task-specific procedure close to the answer.

Tokens, price, and time measure different kinds of efficiency

A harness intended to collect robot experience must make repeated interaction economically practical. The paper analyzes token use, reference API price, and elapsed time over the same 252-episode main cohorts. The three resources do not rank the models in the same order.

Model	RoboDojo Score	Mean tokens / episode	Reference API cost / episode	Time to first reach Score 25
GPT-6 Astra	51.90	16.78M	$21.29	About 42.6 min
Qwen3.8-Flash-Next	31.39	37.67M	$1.09	About 95.5 min
DeepSeek-V4.1-Flash	27.11	48.76M	$0.94	About 99.2 min

Astra reaches a higher Score with fewer tokens, but its reference API cost is far larger. Qwen reaches a higher Score than DeepSeek with fewer tokens for only a modestly higher API price. By the time each model first reaches Score 25, Astra has used about 7.4 million tokens, Qwen 23.5 million, and DeepSeek 49.9 million. Those milestones take 42.6, 95.5, and 99.2 minutes respectively. Qwen's token advantage over DeepSeek therefore does not produce a proportional time advantage, because tool execution, API latency, and physical motion remain major costs.

These are reference API prices, not the full cost of local deployment. Self-hosting also requires GPUs, operations, and idle capacity. Even so, a difference between roughly $1 and more than $21 per episode changes what a research team can afford: thousands of robot episodes, repeated ablations, and retaining failures for later training become practical at one price and prohibitive at another.

What the public release can do today

The project is released under the MIT license as the uni-robot-agent PyPI package, with Codex and Claude Code runtimes. The documented quick start uses Python 3.11 on a Linux GPU host, installs the package, runs uni-robot-agent setup --integration robodojo, and connects to a Responses-compatible model endpoint with an API key. The CLI supports direct and hybrid action-model control. A local dashboard shows camera streams, tool calls, timing, and task outcomes, and saved episodes can be replayed.

Simulation integrations include RoboDojo, RoboUse, and RoboLab. RoboUse connects selected tasks from LIBERO, RoboCasa, MetaWorld, RoboSuite, ManiSkill, Tabletop, Fetch, Kitchen, and several other adapters. RoboLab handles single-DROID tasks in a separately prepared Isaac Sim and Isaac Lab runtime. The Python SDK can run a session against an already running environment and exposes Session.tools for custom strategies. For teams studying tool use, context feedback, or cross-environment trajectory formats, the public simulation stack already supports substantial experimentation.

Physical robots are at a different stage. The paper's real-world evaluation uses an ARX One platform, but the README lists both ARX One and Franka as coming soon and states that the public package does not yet include a supported physical-robot connection workflow. The paper's hardware result should not be read as an install-and-run claim. Real deployment also requires safety limits, real-time control, camera extrinsics, gripper calibration, emergency stops, and recovery procedures that no benchmark score can replace.

What this work actually changes

First, it separates model capability from harness capability. For a robot agent, the model output is only one step in a loop. Whether observations are consistent, actions are idempotent, outcomes are interpretable, and context retains the right evidence all change task success. uni-robot-agent turns those engineering conditions into replaceable system components with a shared tool and feedback format.

Second, it establishes a realistic baseline for open-weight models in robot use. Qwen3.8-Flash-Next reaches 31.39 without depth or verifier feedback in the main simulation setting and 53.50 zero-shot on physical robots. This is not equal to the broad deployment capability of a dedicated VLA, but it moves open models beyond toy demonstrations. With the right interface, they can complete multi-step tabletop manipulation.

Third, it points future training toward four concrete capabilities: understanding scenes, grounding features across views, computing geometry correctly, and choosing physically appropriate interactions. These targets are easier to turn into tasks, rewards, and evaluation dimensions than the vague instruction to collect more robot data. The common interaction record provides a foundation for later supervised or reinforcement learning from the failures and corrections.

The limits also matter. RoboDojo is a controlled simulator, and the physical evaluation contains 19 tabletop dual-arm tasks. OpenRUA uses depth in the hardware comparison while the baselines use RGB. Some baselines retain their published protocols, and the pipeline speed study contains five single-episode configurations. The paper presents a promising research direction and usable infrastructure, not a completed general-purpose robot system.

Conclusion

The main contribution of uni-robot-agent is not the packaging of a model as a robot product. It is evidence that general models need a harness which maintains state, coordinates actions, returns evidence, and permits component replacement before they can enter the physical world. Astra performs strongly in simulation and on hardware, while Qwen shows that open-weight models can be useful at much lower interaction cost. Depth, skill, and sampling analyses make the same point from different angles: performance emerges from the model, information, tools, and feedback together.

The most valuable next step is to use cross-environment interaction records for training open models, while bringing physical connection, precise manipulation, and long-horizon recovery to the same level of reproducibility as simulation research. The current public package is a starting point for simulation research and agent infrastructure. For readers who need a physical robot, the most important missing piece is not another benchmark point. It is a supported, verifiable hardware workflow with explicit safety boundaries.

Resources

• Article: Democratizing Robot-Use Agents: A Unified Harness with Open Models
• Paper: arXiv:2611.01560
• Project: uni-robot-agent.com
• Code and documentation: Open-RUA/uni-robot-agent