CUA-Master: Scaling Computer-use Agents through Open-World Recursive Improvements

Background and motivation

Computer-use agents must interpret a natural-language objective, operate desktop software, web pages, terminals, and cross-application workflows, and continue until an environment verifier confirms completion. Errors compound over long horizons, interface states are often difficult to reverse, and the number of possible application combinations is enormous. The limiting factor is therefore not only model scale, but also the diversity, controllability, and reliability of interaction data.

Conventional pipelines are largely one-directional. Data is collected or synthesized, a model is trained, and benchmark results are inspected afterward. The data builder does not know which operation the next model will fail on, while the training budget is spent without a precise map of persistent gaps. CUA-Master instead treats exploration, data construction, training, and evaluation as an outer loop in which measured weaknesses direct the next round of collection and optimization.

Existing recursive improvement methods can reduce dependence on human labels by training on a model's own experience. Their exploration may still remain closed-world: the policy reinforces familiar distributions and has little mechanism for discovering an unseen app, workflow, or interaction pattern. The authors therefore introduce an external knowledge channel through open-world instructional videos.

Video alone is not supervision. A tutorial often interleaves demonstrations with introductions, advertisements, face-only commentary, slide explanations, cuts, and irrelevant application mentions. The operational signal is temporally sparse, and a reconstructed action may not reproduce the demonstrated state in a real environment. CUA-Master separates the problem into seed-task discovery, dual-path trajectory reconstruction, primitive extraction, and executable validation.

Synthetic tasks produced only by instruction rewriting can vary text while repeating the same latent operation graph. The system instead controls complexity through two explicit variables: the number of composed primitives K and the number of applications N. It also checks preconditions and environment requirements before composition. This makes synthesis auditable, supports difficulty labels, and provides verifiers suitable for reinforcement learning.

Task and action definitions

A task is represented as an instruction, an interactive environment, and a task-specific verifier. The verifier is part of the definition because it determines whether the requested work was actually completed:

$$\mathcal{T}=(\mathcal{I},\mathcal{E},\mathcal{V})$$

Here, I is the instruction, E is the executable environment, and V evaluates the outcome. At decision step s, the policy observes the current state and prior history, then generates one or more atomic actions:

$$\mathbf{a}_{s}\sim\Pi(\cdot|\mathcal{I},o_{s},h_{s})$$

The action batch updates the environment and produces a new observation. A complete trajectory contains the initial observation, the action sequence, and the terminal observation:

$$\mathcal{X}=\{o_1,\mathbf{a}_1,\ldots,o_S,\mathbf{a}_S,o_{S+1}\}$$

CUA-Master uses a hybrid action space. GUI actions cover mouse and keyboard operations, while shell.exec invokes a structured terminal command. Registered functions use tool.call, and finish terminates the episode. Pointer coordinates are normalized to [0,1000], making them independent of the current screenshot resolution.

The public implementation defines GUI, command-line, tool, and control categories and validates action arguments at the schema boundary. Its coordinate contract matches the paper's resolution-independent interface:

# cuamaster/action_space.py
ActionCategory = Literal["gui", "cli", "tool", "control"]

@dataclass(frozen=True)
class CUAAction:
    type: str
    args: dict[str, Any]

    def validate(self) -> None:
        validate_action(self)

def _require_coord(value: Any, name: str) -> None:
    _require_number(value, name)
    if value > 1000:
        raise ValueError(
            f"{name} must be in normalized 0-1000 coordinates"
        )

The split lets the policy choose GUI operations for visual inspection, menus, dialogs, and state confirmation, while shell commands handle batch files, computation, data transformation, and program execution. The experiments show that the two modes complement one another rather than acting as interchangeable alternatives.

The nested recursive flywheel

The outer loop explores the world for applications and workflows, converts executable discoveries into tasks and trajectories, and uses the resulting supervision to improve the model. The updated model produces a new capability profile. Its failures and underrepresented domains then steer the next exploration and data-construction pass.

The inner loop converts external knowledge into data at scale. Open-world seeding supplies authentic but sparse seed tasks. Autocatalytic propagation extracts reusable capabilities from those seeds and composes them into more complex, controllable, and verifiable tasks. The first source preserves real-world grounding; the second expands coverage and raises the density of difficult operations.

flowchart TB
    OW[Open-world videos and applications]
    BF[Evolving model weakness feedback]
    DS[Data construction]
    MT[Model training]

    subgraph INNER[Inner data flywheel]
        SS[Open-world seeding]
        TR[Visual-semantic reconstruction]
        PR[Primitive extraction]
        CP[Constrained composition]
        RB[Snapshot rollback collection]
    end

    OW --> SS
    SS --> TR
    TR --> PR
    PR --> CP
    CP --> RB
    RB --> DS
    DS --> MT
    MT --> BF
    BF --> OW
    MT --> OW

The crucial edge points from training back to open-world exploration. The authors report that later iterations increase the relative representation of software development, science and research, and productivity data while reducing the initial concentration in business administration. The final reinforcement-learning dataset is more balanced across domains.

Open-world seeding

The search space follows a taxonomy containing eight domains, 36 subdomains, and more than 650 applications. Compositional queries combine each application with operation-oriented terms such as tutorial, how-to, and guide. Candidate videos come from YouTube, TikTok, and Bilibili and are deduplicated by source identity and normalized title.

Metadata gates constrain duration, recency, source availability, and taxonomy consistency. A lightweight noise filter removes promotional, entertainment, language-learning, and incidental software mentions. Whisper-Large-v3 through Faster-Whisper produces timestamped transcripts. An annotator then assigns a 0 to 5 CUA-suitability score and predicts whether the video contains a legible computer interaction, a reconstructable trajectory, and sufficient visual clarity.

A single tutorial may contain several workflows. The seed-mining stage combines metadata, transcript, and sampled frames to identify independent temporal episodes. The following mapping expresses its output over episode index, start time, end time, and task definition:

$$\{\hat f_1,\ldots,\hat f_n\}\stackrel{\mathcal A}{\longrightarrow}\{(t_i^s,t_i^e,\mathcal T_i)\}_{i=1}^M$$

Only state-continuous intervals with observable computer operations are retained. Introductions, advertisements, talking-head sections, slide-only explanations, and other non-operational spans are removed. A long video can therefore yield several independently executable tasks, turning a coarse video corpus into task-level supervision.

Visual-semantic trajectory reconstruction

The reconstruction stage runs semantic and visual detection in parallel. The semantic path splits timestamped transcript segments at discourse and temporal markers. It then finds clauses containing action expressions such as click, press, scroll, and drag. The visual path uses FFmpeg frame-level appearance-change scores to detect menu openings, dialogs, navigation, and content updates that may mark a meaningful interface transition.

A candidate boundary must receive support from both channels inside a temporal window. The intersection is written as:

$$\mathcal C=\{(t_i^{\mathrm{sem}},t_j^{\mathrm{vis}})\mid |t_i^{\mathrm{sem}}-t_j^{\mathrm{vis}}|\leq T\}$$

The window size is T. This constraint rejects narration without an interface change and decorative visual motion without an operation. An action rechecker inspects local transcript context and paired pre/post frames, removes non-operational changes, and adds, removes, splits, or merges boundaries.

Accepted candidates are annotated as actions with interaction coordinates, typed text, and other required arguments. A trajectory verifier removes duplicates, splits sequences at substantial temporal or semantic discontinuities, and retains qualified trajectories. Neither a single transcript clause nor a single frame is sufficient evidence.

Primitive extraction and constrained composition

Seeds provide authentic workflows but cannot cover the combinatorial space of computer use. CUA-Master decomposes each seed task and trajectory into application-level ability primitives. A primitive contains the application, a reusable ability, preconditions, environment requirements, and a verifier fragment:

$$(\mathcal T^{seed},\mathcal X^{seed})\stackrel{\mathcal A}{\longrightarrow}\{p_1,\ldots,p_n\},\quad p_i=(u_i,a_i,\mathcal P_i,\mathcal E_i,\mathcal V_i)$$

The primitive library canonicalizes and deduplicates semantically equivalent abilities. Variants remain separate when their preconditions, environment requirements, or verification logic differ. The representation abstracts away concrete files and content without discarding the dependencies needed for reliable execution.

Given a complexity budget K and an application set, the composer builds compatible primitive sequences and synthesizes a coherent instruction, executable environment, and task-level verifier:

$$(\mathcal B(\{u_1,\ldots,u_N\}),K)\stackrel{\mathcal A}{\longrightarrow}\mathcal T^{syn}=(\mathcal I^{syn},\mathcal E^{syn},\mathcal V^{syn})$$

With N=1, the result is a single-application task; with N>1, it is a cross-application task. Composition first checks preconditions and environment requirements. Conflicting capabilities cannot be joined, and the synthetic precondition is the conjunction of compatible primitive conditions.

The code defaults to 2 to 4 primitives per variant and exposes four composition strategies: sequential, parameterized, state-dependent, and cross-application handoff. It simulates effects in execution order and checks whether the environment supports the required VM capabilities:

# cuamaster/propagation/generation/composer.py
PrimitiveComposer(
    min_primitives=2,
    max_primitives=4,
    strategies=(
        "sequential",
        "parameterized",
        "state_dependent",
        "cross_app_handoff",
    ),
)

required = {
    item
    for capability in capabilities
    for item in capability.required_vm_capabilities
}
if not environment.supports(sorted(required)):
    return False

The primitive extractor adds two source-grounding gates from the repository. It discards a capability when no source evidence exists and when the selected environment does not support its VM requirements. This prevents freely generated abstractions from entering the training set without evidence or an execution path.

Snapshot rollback and reliable collection

When a long rollout fails, most of the preceding actions may still be valid. Restarting from the initial state wastes that verified prefix, while accepting the failed suffix introduces inconsistent supervision. A detector examines the verifier residual, trajectory, and executable environment to estimate the earliest state-divergence step and produce grounded feedback:

$$(\mathcal T,\mathcal X,\mathcal E)\stackrel{\mathcal A}{\longrightarrow}(\hat s,\mathcal F)$$

The feedback states why execution failed or became inefficient but does not expose a privileged reference action. The environment rolls back to the latest valid snapshot before the estimated divergence step. The policy keeps the stable prefix, discards the bad suffix, and resumes with the diagnostic hint.

Snapshots are incremental and created at stable boundaries. Each stores VM state plus a lightweight state signature covering active applications, window layout, relevant files, and controller metadata. This avoids replaying the entire history from an initial image. Retries continue until the verifier passes or the budget expires.

This process acts as a branching search that reuses successful prefixes. At 40 primitives across three applications, CUA-Master reaches a 49.8% rollout success rate while a baseline that restarts from the beginning with the same retry budget reaches 14.6%. At the extreme 50-primitive, eight-application setting, the numbers are 18.7% and 0.0%.

Training recipe

Supervised fine-tuning first builds broad operational competence. The final iteration then applies reinforcement learning to improve long-horizon decisions. For SFT iteration i, the policy trains on the cumulative union of seed and synthetic trajectories from earlier iterations:

$$\mathcal D_i^{SFT}=\bigcup_{t=1}^{i}(\mathcal X_t^{seed}\cup\mathcal X_t^{syn})$$

Every SFT iteration starts from the same base checkpoint to limit optimization drift, while each updated dataset adds new trajectories to the original set. The three SFT additions contain 63K, 38K, and 14K tasks with trajectories, producing about 115K SFT samples. Another 12K high-quality tasks are used for RL, for roughly 127K total samples.

The base model is Qwen-3.5-27B. SFT uses AdamW, a peak learning rate of 5e-6, a cosine schedule, a 0.03 warmup ratio, and batch size 128. RL uses a modified ScaleCUA/GRPO-style optimizer with learning rate 1e-6 and batch size 64. Each task samples eight trajectories in real VMs, allowing up to 512 trajectories per batch.

Training runs on 16 nodes, each with eight NVIDIA H200-141GB GPUs. The VM infrastructure spans Linux, Windows, and macOS, contains more than 100 environments, and supports over 2,000 concurrent virtual machines. Domain-level and application-level failures plus rollout success rates determine the next iteration's data scale and difficulty.

Main experimental results

The evaluation uses two GUI-only benchmarks, OSWorld Verified and OSWorld 2.0, plus two hybrid GUI/CLI benchmarks, WeaveBench and Agents' Last Exam. Agents' Last Exam is measured by full-pass rate. Baselines include general models from Qwen, Minimax, and Kimi, and specialized systems such as OpenCUA, GUI-Owl, EvoCUA, ScaleCUA, Qwen-CUA, and UI-Mate.

CUA-Master-27B scores 80.2 on OSWorld Verified, 36.9 on OSWorld 2.0, 43.9 on WeaveBench, and 25.0 on Agents' Last Exam. Relative to the similarly sized UI-Mate-27B, the gains are 2.6, 12.9, 25.6, and 11.8. The gap expands most on the two hybrid benchmarks, where CLI execution can remove long GUI-only interaction sequences.

Benchmark	Mode	CUA-Master-27B	UI-Mate-27B	Gain
OSWorld Verified	GUI only	80.2	77.6	+2.6
OSWorld 2.0	GUI only	36.9	24.0	+12.9
WeaveBench	GUI + CLI	43.9	18.3	+25.6
Agents' Last Exam	GUI + CLI	25.0	13.2	+11.8

The OSWorld Verified domain breakdown shows 91.6 on operating-system tasks, 88.0 on professional tasks, and 83.1 on office tasks. Workflow is the weakest domain at 70.7. The profile suggests strong local and single-application operation but persistent difficulty when a task requires coordination across several steps or applications.

OSWorld Verified domain	Tasks	Score	Interpretation
Office	117	83.1	Stable on a large office subset
Professional	49	88.0	Strong specialist-software operation
Workflow	93	70.7	Long multi-step work remains hardest
Daily	78	78.8	Strong completion on routine software
OS	24	91.6	System-level operations near saturation
Overall	361	80.2	Competitive aggregate specialist result

OSWorld 2.0 is substantially harder, lowering the aggregate to 36.9. Performance varies from 44.7 in research and education to 29.4 in creative production. On WeaveBench, Web Frontend reaches 58.4 and Desktop OS 55.3, while Spatial/CAD reaches only 25.9 and Design 26.3. Professional tools with specialized spatial or visual state remain a clear boundary.

Efficiency, platform transfer, and action modes

CUA-Master completes OSWorld Verified in eight minutes on average. On OSWorld 2.0, it slightly exceeds Qwen-3.8-27B, 36.9 versus 36.5, while reducing runtime by 24%. WeaveBench takes 34 minutes and matches GPT-5.6-Luna with 28% less runtime. On Agents' Last Exam, it matches Kimi-K3 and Qwen-3.8-Max with 34% to 36% less time.

Cross-platform tests examine whether the action representation transfers beyond Linux. The model scores 56.2 on WindowsAgentArena, 43.0 on MacOSWorld, and 59.9 on MacAgentBench. The Windows result is below the strongest specialized baseline, but the two macOS results show that the open-world data and common action interface transfer across operating systems.

Cross-platform benchmark	CUA-Master-27B	Qwen-3.5-27B	Qwen-3.8-27B
WindowsAgentArena	56.2	40.7	54.4
MacOSWorld	43.0	30.4	39.2
MacAgentBench	59.9	44.3	56.6

The action-mode study provides the clearest mechanism evidence. On WeaveBench, hybrid execution scores 43.9, compared with 6.6 for GUI only and 11.5 for CLI only, and uses 23.8% fewer steps than GUI only. On Agents' Last Exam, hybrid execution raises the score from 5.9 to 25.0 and reduces steps by 26.7%. OSWorld 2.0 reduces steps by 43.8% in hybrid mode.

Benchmark	GUI only	CLI only	Hybrid	Hybrid step change
WeaveBench	6.6	11.5	43.9	23.8% fewer than GUI only
Agents' Last Exam	5.9	Not reported	25.0	26.7% fewer than GUI only
OSWorld 2.0	Not reported	Not reported	36.9	43.8% fewer steps

These differences are not evidence that CLI is universally stronger. CLI-only execution loses visual grounding and can fail when the task depends on menu state, dialogs, or visual confirmation. GUI-only execution wastes steps on batch processing, structured transformation, and programmatic calls. Hybrid performance comes from choosing an abstraction level that matches the current subtask.

Evidence for the data flywheel

A t-SNE view of task embeddings compares CUA-Master tasks with AgentNet, VideoCUA, ScaleCUA, WildGUI, CUA-Gym, and ProCUA-SFT. The generated tasks overlap existing distributions, indicating that composition stays close to executable computer-use semantics, while also extending into sparsely covered regions. Both overlap and extension are necessary: pure novelty can be meaningless, while pure overlap adds little capability.

Rollback benefits grow with composition difficulty. On five primitives across one application, the restart baseline succeeds 74.0% of the time and CUA-Master reaches 95.4%. At 40 primitives across three applications, the baseline falls to 14.6% while CUA-Master remains at 49.8%. At 50 primitives across eight applications, the baseline records no success and CUA-Master still reaches 18.7%.

Primitives / applications	Restart baseline	CUA-Master	Absolute gain
5 / 1	74.0%	95.4%	+21.4
10 / 1	53.0%	89.1%	+36.1
20 / 1	31.5%	83.7%	+52.2
10 / 3	29.2%	62.3%	+33.1
20 / 3	19.3%	56.9%	+37.6
40 / 3	14.6%	49.8%	+35.2
20 / 5	1.9%	32.8%	+30.9
40 / 5	0.4%	24.4%	+24.0
50 / 8	0.0%	18.7%	+18.7

Near-zero baseline success on the hardest settings shows that additional sampling alone cannot solve structured long-horizon failures. Reusing a valid prefix and restarting from the latest stable state changes the yield of the same rollout budget.

The recursive training curve reports gains above 25 points on OSWorld Verified and WeaveBench. Once SFT improvements on easier tasks approach saturation, the final RL stage adds roughly 15 points on OSWorld 2.0 and Agents' Last Exam. Stronger annotators produce larger gains, while a self-policy annotator improves results only modestly.

Why the method matters

The main contribution is not a single generator. CUA-Master places source discovery, execution structure, optimization, and capability feedback in one loop. Open-world seeding supplies external knowledge, primitive extraction makes that knowledge composable, environment verifiers make tasks testable, and snapshot rollback increases trajectory yield on difficult combinations.

The structure is also relevant to embodied foundation models. Vision-language-action systems need demonstrations, reusable skills, simulator or robot verification, and local recovery after failure. The paper treats data generation as a grounded system involving external experience, executable constraints, and feedback, rather than an offline prompt-only module.

For computer-use products, the hybrid action result is a design argument for selecting the right abstraction level. GUI actions expose the same state a person sees; CLI actions give exact batch and programmatic control. A policy that can move between them avoids excessive visual grounding while retaining visual confirmation when the task requires it.

Limitations

The authors report that recursive improvement requires substantial infrastructure for environment interaction, trajectory collection, and iterative model optimization. Snapshot rollback reduces repeated work but does not remove the VM fleet and orchestration burden. The experiments use 16 nodes with eight H200-141GB GPUs per node and support more than 2,000 concurrent VMs, making exact reproduction expensive.

The authors also report that the quality ceiling of synthetic data depends on capable annotator models. Stronger annotators consistently yield larger gains, which introduces inference cost and limits accessibility when such models are unavailable. Self-policy annotation helps, but its gains are smaller.

An independent limitation concerns composed verifiers. Several local checks may all pass while the cross-application result is semantically wrong. Composition can also produce checks that are too weak, redundant, or too strict for legitimate shortcuts. The release gate addresses quality and deduplication, but the paper does not provide systematic false-accept and false-reject rates for verifier composition across domains.

Evaluation coverage is another boundary. The main experiments focus on desktop applications and operating systems. Mobile interaction, multi-tab browser collaboration, enterprise permission systems, real-time collaborative software, and workflows that depend on physical sensors or organizational processes are largely outside the measured distribution.

Conclusion

CUA-Master links open-world videos, reusable primitives, verifiable tasks, reliable trajectories, and model training through a nested recursive system. The loop is not confined to self-generated experience: external applications and workflows continuously add knowledge, while failures on the current policy determine where the next iteration spends data and compute.

The 27B model's results on four main benchmarks and three cross-platform tests show that data structure and execution mechanics can matter as much as parameter count. Visual-semantic reconstruction, constrained primitive composition, hybrid GUI/CLI control, and snapshot rollback address distinct bottlenecks in supervision and rollout. Lower infrastructure cost, stronger cross-primitive verifier semantics, and broader mobile or enterprise coverage are the main paths toward a more sustainable general computer-use training system.