#Multimodal UI (1)

Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents

The paper introduces Hybrid Computer Use (HCU), a unified agent interface that combines code and command-line interaction, accessibility-tree semantics, and screenshot-based mouse and keyboard control in one context. Compact observations, batched action execution of up to 64 steps, wait steps, post-action diffs, and cross-channel goal and layout guards reduce round trips and prevent stale-input actions. HCU improves task success or interaction efficiency across OSWorld 2.0, OSWorld 1.0, WeaveBench, macOS, and Windows Agent Arena.