Computer use

Instead of calling purpose-built APIs, a computer-use agent perceives the screen through screenshots and acts through the same primitives a human uses: moving the cursor, clicking, typing, scrolling. That generality lets it drive any software (legacy systems, web apps without APIs, desktop tools) which is precisely the work that resists conventional automation.

Anthropic shipped the first frontier computer-use capability in late 2024, and rivals followed with browser-operating agents and dedicated computer-use models; by 2026 the capability had matured from fragile demos into products handling real multi-step workflows, with browser automation the most reliable slice.

It is among the most expensive capabilities per task: each loop iteration sends a fresh screenshot (an image worth a thousand or more input tokens) and nontrivial tasks take dozens of iterations. Sandboxing, confirmation gates for irreversible actions, and defences against prompt injection from on-screen content are standard deployment practice.