Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents

Paper: Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents
Authors: Haoqing Wang, Xiaoyi Zhang, Sicheng Fan, Zhexin Zhang, Xinjie Zhang, Jiahao Li, Bin Li, Yan Lu
Paper: https://arxiv.org/abs/2610.41432
Code: Public and MIT licensed, microsoft/Hybrid-Computer-Use
Project page: https://microsoft.github.io/Hybrid-Computer-Use/
Version: arXiv v1, October 30, 2026
Agientry facet: Harness / Development

Why screenshot-first computer use becomes a bottleneck

Computer-use agents have largely inherited the human desktop loop: inspect a screenshot, locate a control, move the pointer, and type. The interface is broad and intuitive, but it forces a capable language model to translate semantic intent into repeated pixel interpretation and coordinate actions. A task that could be completed by reading a field or editing a file often begins with a visual search for the correct control.

The cost grows with OSWorld 2.0 style work. Long-horizon professional workflows cross applications, files, dialogs, browsers, and document editors. Every new screenshot can invalidate prior assumptions, while every extra observation consumes context and creates another chance to misidentify an object. Reducing the number of round trips becomes as important as improving a single click.

The paper separates three interaction channels. Programmatic interaction uses code, shell commands, and file tools. Semantic interaction reads the accessibility tree and addresses controls by stable references. Visual interaction uses screenshots, zoom, pointer input, and keyboard input. These channels expose different evidence and different failure modes, so the research question is not which one wins. It is how a general agent can select and combine them without acting on stale state.

Earlier hybrid systems often add code to a visual loop or use accessibility data mainly to locate coordinates. Some split work among an orchestrator and specialized GUI agents. HCU takes a narrower system position: expose all three channels as tools in the existing agent loop, add cross-channel guards, and let the model decide where each task should be executed.

This makes the work a study of the agent-computer interface. It does not introduce a new foundation model, planner, or training regime. It changes what the model can observe, how it references a target, how many actions can be composed, what evidence comes back, and when an earlier observation must be rejected.

What each channel contributes

The programmatic channel is deterministic and composable. It can inspect files, transform structured data, run tests, and verify persistent artifacts. It is not limited by whether a UI framework exposes a control. Its weakness is coverage: some professional applications expose little more than a graphical surface, so code cannot perform every user action.

The semantic channel exposes roles, names, values, states, and supported operations from the application accessibility tree. A button remains a button rather than a rectangle, and a target can be identified without knowing its screen coordinate. The channel is incomplete by construction. Applications choose what to expose, custom-drawn interfaces may omit elements, and virtualized lists may not create an item until it is rendered.

The visual channel provides the final observable surface. It can show layout, rendered content, custom graphics, warnings, and visual effects that are absent from the semantic tree. It cannot prove semantic identity by itself. Two identical-looking buttons may perform different actions, and an old coordinate can point to a different control after a window moves.

HCU does not treat the semantic and visual channels as a fixed hierarchy. The same model can use a screenshot to understand a visual result, then use an accessibility reference to execute the next action. It can also use code to inspect a file and a screenshot to check how the application presents that file. Observation and execution do not need to use the same channel.

Method: one interface, two action guards, and an explicit evidence contract

1. Three channels inside the existing harness

HCU adds an accessibility tool and a graphical tool through the Model Context Protocol. The host agent keeps its existing programmatic tools and execution loop. Both desktop tools act on the same computer, and all results return to the same model context. No additional planner, subagent, or task controller is inserted between the model and the interface.

The design supports GPT-6 Astra in Codex, Claude Opus 5 in Claude Code, and several other model and harness combinations. Because the underlying harness remains in place, comparisons can focus on the interaction surface rather than on unrelated planning features. The paper disables web search, memory, subagents, and planning modes during matched comparisons so both interfaces share the same base policy and programmatic tools.

flowchart LR
    T[Task] --> H[Existing agent harness]
    H --> P[Programmatic tools]
    H --> S[Semantic AX MCP]
    H --> V[Visual GUI MCP]
    P --> C[Shared desktop state]
    S --> TG[Target guard]
    V --> LG[Layout guard]
    TG --> A[Batched actions and waits]
    LG --> A
    A --> E[Effect report and AX diff]
    E --> H
    C --> TG
    C --> LG

The unusual part of this architecture is the loop back from live desktop state into the guards. The agent does not carry a permanently valid world model. Each action checks the current object identity or layout lease, and a mismatch produces evidence instead of silently redirecting the action.

2. Compact semantic views with references bound before compression

Raw accessibility trees contain structural wrappers and repeated nodes that are expensive to put in context. HCU keeps the full acquired records for identity checks and state comparison, but returns a compact view to the model after removing wrappers and condensing repetition. The public target reference is bound to the underlying element before the view is shortened, so compression does not change which object a reference denotes.

Targets omitted from the compact view can be recovered through a search or a scoped observation. This gives the model a cheap default representation and a precise fallback without forcing every decision to carry the entire tree.

The accessibility action mode can compose up to 64 ordered steps. A wait step emits no input and holds until a condition appears, such as a window, target reference, text, query result, or stable interface. Waits can sit between mutating steps to prevent later actions from running in the wrong state, or at the end to validate the result. All waits in one call share a 60 second budget.

If an action fails, a wait times out, or window continuity is lost, the batch stops and reports where execution ended. The model receives a continuation point and evidence rather than a generic failure. This avoids blind replay and reduces the number of separate observations used only to ask whether the previous action worked.

3. Semantic target guard

Cross-channel work creates a stale-evidence problem. An accessibility action may open a dialog that covers coordinates from an earlier screenshot, while a graphical action may replace an element that a semantic reference previously denoted. HCU does not look for a similar target when the original identity is missing. It fails closed and asks the model to observe again.

For every semantic action, the provider resolves the reference against a live target and checks whether that target supports the requested operation. The simplified admission rule is:

$$\mathrm{ok}_{AX}=(t\neq\bot)\land\mathrm{CanExecute}(a,t)$$

Here $a$ is the attempted action, $t$ is the live target, and $\bot$ means that identity cannot be established. The equation is deliberately simple: when identity or capability is uncertain, no input is emitted. A missing element, ambiguous match, replaced element, or unsupported operation becomes a precondition failure.

The guard also runs on every action inside a large batch. Earlier steps can change the window or replace later targets, so checking only the first reference would leave the same stale-state risk at a smaller scale. Each step must remain valid immediately before its own input.

4. Graphical layout guard

Coordinate actions cannot be validated by looking only at the point that will be clicked. The point may contain a different window after focus changes, a dialog may move above it, or the application may resize. HCU associates coordinate evidence with a screenshot frame and compares the current desktop layout before dispatching guarded input.

The layout sample includes desktop bounds, visible top-level windows, stacking order, active window, and the window under the target point. The admission condition can be summarized as:

$$\mathrm{ok}_{GUI}=(F\neq\bot)\land(\mathrm{LiveLayout}()=F.\mathrm{layout})$$

$F$ is the screenshot frame on which the coordinates depend. If the frame is unavailable, or if the freshly sampled layout no longer matches, the batch fails before input. The response includes fresh visual evidence so the model can choose new coordinates instead of repeating the old request.

The repository implements this behavior directly. In the Linux dispatch path, a changed frame is rejected with the public code stale_gui_frame when the old coordinates no longer match the live layout. The result sets dispatched input and emitted input events to zero, making the failure observable to both the model and the surrounding harness.

5. Feedback tied to evidence, not an optimistic success flag

After an accessibility action, HCU returns per-target effect summaries, window changes, and a diff against the last eligible observation. The model can ask for a screenshot, execution evidence only, or a fuller observation. Returning a diff with the action often supplies enough information for the next decision without another round trip.

The interface distinguishes known non-delivery from an unknown effect after possible input. It also distinguishes an invocation from a confirmed state change. This vocabulary matters because accessibility APIs can report that an action was invoked even when the application did not reach the intended state. The model is told what the evidence supports and must decide whether to retry, inspect, or stop.

The same evidence contract appears in error responses. A guard rejects a stale target before input, while a possible post-input failure may be marked as effect unknown. The agent should not treat those cases as equivalent. One is safe to recover by refreshing state; the other may require product inspection before retrying a mutating action.

6. Skill instructions are part of the interface

HCU includes a supplementary skill that explains channel selection, target references, batching, waits, verification, and recovery. It is not a set of task-specific answers. It describes the contracts that are hard to infer from raw tool schemas: when a screenshot is insufficient, how to prove that a file was saved, what to do after a stale reference, and how to avoid malformed calls.

The skill ablation shows that these instructions are not cosmetic. The proportion of accessibility and graphical calls rejected for malformed arguments fell from 17.37% to 1.36%. Task-runs with consecutive schema failures fell from 44.69% to 0.37%. The skill also increased the use of multi-action accessibility batches, targeted search, and in-call checks while reducing separate observations.

This result is useful for harness design beyond HCU. A tool surface and its usage policy form one product. If the policy is missing, the model may have the necessary capability but fail repeatedly at the boundary.

7. The published source matches the paper's guards

The semantic resolver in the public repository checks whether the live record still exists and still belongs to the expected identity. If not, it raises a structured stale-reference failure before input:

node = record.pop("_resolvedNode", None)
if node is None:
    node = find_element_at_snapshot_path_with_bounds(
        app, record, matched_bounds)
if node is None:
    raise OperationFailure(
        "stale_element_ref reason=live_semantic_record_changed_before_input "
        "no input was emitted; refresh AX mode=observe/find and use the "
        "newly returned public ref.",
        failure_code="stale_element_ref",
        failure_phase="precondition",
        input_events_emitted=False,
    )
return node

Source: computer-use/native/linux/provider/element_resolver.py:488-500. The error includes a public failure code, a precondition phase, and the fact that no input event was emitted.

The graphical dispatcher performs the corresponding frame check. When the current capture differs from the reference frame and the old coordinates no longer match the live layout, it returns a stale-frame result with no dispatched actions:

reference_layout = _published_layouts.get(frame_id) if frame_id else None
if frame_id and frame_guarded_actions:
    if (current_capture.frame_id != frame_id
            and not coordinates_still_valid(
                reference_layout, capture_layout(current_capture), validated)):
        return {
            "ok": False,
            "error": "The desktop layout changed since the screenshot used "
                     "for coordinates. No actions were dispatched and no input "
                     "events were emitted.",
            "failureCode": "stale_gui_frame",
            "failurePhase": "precondition",
            "dispatched": 0,
            "input_events_emitted": False,
            "snapshot": encode_evidence(current_capture),
        }

Source: computer-use/native/linux/provider/gui_dispatch.py:572-591. The implementation returns fresh evidence rather than automatically finding a similar coordinate target.

Figure 13: The HCU tool surface connects through MCP to native backends, with guards placed before real input.

Figure 17: Cross-channel state and action guards constrain both semantic references and visual coordinates.

Figure 18: A Writer layout change produces a fresh-evidence rejection instead of silently reusing old coordinates.

Experiments: success improves while interaction cost falls

The main evaluation uses all 108 OSWorld 2.0 tasks with a 500-step limit. Partial is the mean task score and Binary is the full-credit rate. In the matched GPT-6 Astra comparison, CUA Driver reaches 68.82% Partial and 31.50% Binary. HCU reaches 72.37% and 32.40%, respectively.

The important part is the accompanying resource profile. HCU improves Partial by 3.55 points while using 29.81% fewer tool calls, 9.57% fewer Steps, and 21.00% less mean time. Cost is essentially unchanged at USD 10.03 versus USD 10.05. The interface does not buy accuracy by spending more operations; it obtains more usable information and execution progress per round trip.

Claude Opus 5 provides a second model result. HCU reaches 72.06% Partial and 35.19% Binary, improving over the official leaderboard result by 3.75 and 3.76 points while reducing cost from USD 28.43 to USD 22.67, a 20.26% decrease. The interface therefore transfers across two substantially different harnesses and action-vocabulary conventions.

OSWorld 1.0 extends the comparison to four more model and harness combinations. Across GPT-5.4, Claude Opus 5.5, Gemini 3.8 Flash, and Grok 4.7, HCU averages 6.48 more score points, 17% fewer Steps, and 30% lower cost than CUA Driver. The result is not a promise that every task benefits equally, but it rules out a gain tied to only one model or one agent runtime.

The released version also scores 4.36 points above the evaluated version while using fewer Steps and less cost. The authors attribute part of this change to interface refinements: ordered waits, a larger 64-step batch limit, public error codes, graphical zoom, per-target effects, and point-level layout checks on Linux. This is evidence that harness revision itself can change benchmark outcomes.

WeaveBench tests longer hybrid workflows. On all 114 tasks, HCU scores 53.25%, compared with 46.37% for CUA Driver and 43.84% for the official OpenClaw baseline. It doubles the number of tasks scoring at least 0.8 against CUA Driver, from 12 to 24, while using 17.6% fewer tool calls, 48.6% fewer input tokens, and 46.3% less cost per task.

The multi-platform evaluation tests whether the abstraction survives outside Linux. On 28 UI-dependent macOS tasks, HCU completes 25 versus 24 for CUA Driver and cuts average time from 876.0 to 544.5 seconds. On 111 Windows Agent Arena tasks, score rises from 71.89% to 80.86%, average Steps fall from 24.45 to 17.35, and total cost drops from USD 73.00 to USD 36.06.

The channel ablation gives a more specific account of where the gains come from. Removing accessibility lowers GPT-5.4 from 65.59% to 62.79%, and full HCU leads in every one of the three runs. Removing the skill lowers the mean to 64.00%, with greater run-to-run variance. These are consistent success effects rather than a single lucky run.

The graphical ablation needs a careful interpretation. Removing GUI produces a 66.08% mean with a 1.66 point standard deviation, while full HCU has 65.59% with a 0.61 point deviation. The score comparison changes sign across runs, but GUI consistently reduces Steps by 12.45%. In this benchmark, the strongest defensible claim is that GUI improves interaction efficiency and supplies visual evidence, not that it creates a stable accuracy gain.

Why this matters for agent harnesses

HCU reframes computer use as interface design. For a fixed model, changing what is observable, addressable, composable, and verifiable can materially change task performance and cost. The semantic channel exposes application structure, the programmatic channel removes repeated manual work, and the visual channel supplies evidence that neither of the other two can guarantee.

The trajectory analysis also shows that models do not use the same interface in the same way. Claude uses roughly 3.3 times the Steps of GPT-5.4 and much more of every tool category. Two agents can have similar overall Step counts while displaying very different channel mixes. Harness evaluations should therefore report interaction structure, success, and efficiency together.

The guard pattern generalizes beyond desktop control. Browser references, database rows, robot end-effectors, and remote objects can all outlive the observation that justified an action. A robust system needs an evidence lease, a live precondition check, and a fail-closed result that tells the caller how to recover. Treating a stale reference as a retryable transport error is insufficient.

For embodied systems, the analogy is not that robots should expose desktop accessibility trees. The transferable idea is layered abstraction. Code can handle deterministic planning and data transformation, semantic state can expose known objects and legal operations, and vision can handle open-ended evidence and anomalies. The model should be able to compose those layers while each actuator boundary checks its own preconditions.

The paper also keeps action delivery separate from task completion. It does not claim that a successful click or accessibility invocation proves the requested artifact exists. The model must combine file checks, semantic state, interface diffs, and visual evidence. This separation is important for production agents because optimistic tool success flags are a common source of false completion.

Limitations and independent assessment

The authors explicitly note that accessibility coverage depends on the application. Custom-drawn content may have no accessible elements, accessible objects need not map one-to-one to visible widgets, and virtualized items may be absent until rendered. HCU keeps visual observation because semantic coverage cannot be assumed.

The authors also state that an accessibility invocation does not prove the target state changed. Semantic evidence cannot prove pixels, persistent file contents, or overall task completion. Visual evidence cannot prove semantic identity or persistence. Cross-channel evidence still requires interpretation by the model.

Model-specific action conventions remain a limitation. In the GUI vocabulary experiment, Claude Opus 5.5 improves by 0.82 points with Claude-style actions, while GPT-5.4 loses 2.03 points when moved away from the default GPT-style vocabulary. A harness can expose the same capabilities but still favor the native calling convention of a particular model.

From an engineering perspective, HCU depends on native providers, desktop identity, screenshot leases, and window continuity. The paper evaluates isolated desktops and task completion; it does not settle multi-user concurrency, malicious accessibility spoofing, sensitive-field redaction, least privilege, or revocation. Those controls need separate design before enterprise deployment.

Final task verification is another open area. HCU reports local effects and state differences, but the model still decides whether a spreadsheet contains the right formulas, a document satisfies a formatting requirement, or two applications agree on the same record. External validators, database queries, and application-native checks could become additional evidence sources in the same harness.

Conclusion

HCU changes the contact surface between agents and computers rather than the underlying foundation model. It combines programmatic, semantic, and visual interaction, compresses semantic observations, batches actions with waits, returns useful state differences, and guards every input against changed targets or layouts.

The results span OSWorld 2.0, OSWorld 1.0, WeaveBench, a macOS UI subset, and Windows Agent Arena. Across these settings, HCU improves task performance, interaction efficiency, or both, and it transfers across Codex, Claude Code, Gemini CLI, and Grok Build. The broader lesson is not a single benchmark number. It is that a capable model should not be forced into a human-only action space when the computer already exposes richer interfaces.

For Agientry's Harness facet, the paper offers concrete design rules: define tool contracts together with state evidence, stop before input when identity is stale, make recovery instructions part of the interface, and measure efficiency alongside success. Long-horizon agents need all four properties to become dependable production systems.

The official code is public under MIT, with architecture documentation, MCP definitions, a skill, and Linux, macOS, and Windows backends. Teams that reuse the ideas should first reproduce the guard and evidence contracts on their own trajectories, then decide which tool surfaces and permissions are safe for production.