Computer Use Agents Explained: How AI Operates a Desktop

Ask a developer to automate a task that lives entirely inside a legacy web portal or a Windows desktop app with no API, and until recently the honest answer was robotic process automation: scripts tied to fixed pixel coordinates that break the moment a vendor moves a button. Computer use agents replace that script with a model that watches the screen and decides what to click.

Computer use agents are AI systems that take a screenshot of a computer screen, reason about what is on it, and issue mouse and keyboard actions to complete a task, the way a person would use a browser or a desktop app. Unlike a tool-calling agent that hits a documented API, a computer use agent works through the same graphical interface a human sees, so it can operate software that has no API at all.

How it works

Every computer use agent runs the same loop, described in OpenAI's original Computer-Using Agent announcement: perceive, reason, act. A screenshot is added to the model's context so it has a visual snapshot of the current screen. The model reasons through the next step with chain of thought, taking prior screenshots and actions into account. It then performs an action, clicking, typing or scrolling, and the loop repeats until the task is done or the agent needs a person's input, such as entering a password or solving a CAPTCHA.

Toloka's explainer breaks the same loop into three parts: a perception engine that turns a screenshot into a structured map of buttons, fields and menus; a reasoning engine, usually a multimodal LLM, that plans the next sub-step and adapts when something unexpected happens; and an action interface that moves the cursor, types and switches windows at the operating-system level. This is what separates a computer use agent from RPA: RPA memorizes pixel coordinates and fixed rules, so it collapses the moment a layout changes, while a computer use agent reads labels and icons for what they mean and keeps working after a redesign.

The major model providers expose this loop as a tool call rather than a separate product. Anthropic's computer use tool is a client toolset, computer_toolset_20260801, that gives Claude 17 member tools such as screenshot, left_click, type and zoom in a single tools entry, and the application running the loop supplies the environment, usually a sandboxed virtual machine or container. Anthropic's documentation is explicit that computer use carries risks distinct from other API features, and recommends a dedicated VM with minimal privileges, no access to sensitive account credentials, an allowlist of reachable domains, and a human confirmation step before sensitive actions. Earlier tool versions, including computer_20251124, keep working for models that do not support the newer toolset. For tasks confined to a single webpage, Anthropic offers a separate browser use tool that acts on the page directly instead of a full desktop.

Microsoft takes the same model-as-a-tool approach inside Copilot Studio. An agent's computer use tool runs against a Windows machine the maker configures, and the maker picks the underlying model from a table Microsoft maintains: OpenAI's Computer-Using Agent (Standard tier, generally available), Claude Sonnet 4.5 (Standard, generally available), Claude Sonnet 4.6 (Standard, experimental) and Claude Opus 4.6 (Premium, experimental). Credentials for logging into target sites and apps are stored either in Power Platform's internal storage or in an Azure Key Vault the maker supplies, and a human supervisor can be notified by email if the agent detects instructions that look like they are trying to alter its behavior.

Examples

The tool landscape splits into open source frameworks and proprietary products, catalogued in Turing Post's August 2026 comparison. On the open source side: UI-TARS, ByteDance's family of GUI-control models with a companion desktop app; Agent S3, which runs on Linux, macOS or Windows and can drop into Python or Bash when a step is easier to script directly; Browser Use and Stagehand, both browser-only frameworks under the browser automation category that developers wire up to their own choice of model; Skyvern, licensed AGPL-3.0, which can trigger a source-disclosure obligation if a hosted version is modified and offered over a network; OpenAdapt; and UFO³, Microsoft's own open source agent for cross-device workflows.

On the proprietary side, OpenAI's Computer-Using Agent powers Operator, released as a research preview restricted to ChatGPT Pro users in the US. Anthropic ships Claude Cowork, which InfoQ reported Anthropic announced on 12 January 2026 as a macOS agent for file management and document processing, built on top of the same computer use capability. Other proprietary entries in Turing Post's table include Gemini in Chrome, Perplexity's Personal Computer assistant for macOS, Amazon Nova Act for browser automation, and Manus's Browser Operator.

OpenAI's own numbers from the CUA launch show what these models could do at release: 38.1% on the OSWorld full-computer-use benchmark, 58.1% on WebArena and 87% on WebVoyager for web tasks, against a human OSWorld baseline of 72.4%. a16z's August 2026 research tracks how far the field has moved since: on OSWorld-Verified, the best model scored 42% a year before their report and 85% at the time of writing, above the roughly 72% human testers score on the same tasks, with Claude Fable 5 cited as the current leader. One founder they quote put the production threshold at a specific point: