Computer use

Instead of calling purpose-built APIs, a computer-use agent perceives the screen through screenshots and acts through the same primitives a human uses: moving the cursor, clicking, typing, scrolling. That generality lets it drive any software — legacy systems, web apps without APIs, desktop tools — which is precisely the work that resists conventional automation.

Anthropic shipped the first frontier computer-use capability in late 2024, and rivals followed with browser-operating agents and dedicated computer-use models; by 2026 the capability had matured from fragile demos into products handling real multi-step workflows, with browser automation the most reliable slice.

It is among the most expensive capabilities per task: each loop iteration sends a fresh screenshot — an image worth a thousand or more input tokens — and nontrivial tasks take dozens of iterations. Sandboxing, confirmation gates for consequential actions, and defenses against prompt injection from on-screen content are standard deployment practice.