Computer Use
An agent operating a computer the way a person does — through the screen, mouse and keyboard — and the hardest reliability problem in agents.
Reviewed July 16, 2026Stable
When not to use it
- Where an API exists. Always. It is faster, cheaper, deterministic, and doesn't break on a redesign.
- High-frequency automation. The per-step cost multiplies by volume and the failure rate compounds by length; both go the wrong way.
- Anything irreversible without a human gate. Misclicks don't throw exceptions — they perform actions.
Reach for something else instead
- The API, if one exists, ends the conversation.
- MCP or a tool interface exposes the function directly instead of teaching an agent to find its button.
- RPA — traditional robotic process automation — is more brittle but deterministic and far cheaper per run, and for a fixed, unchanging workflow it's often the right answer.
- A script. Most computer-use demos automate something
curldoes.
Sources & further reading
- Xie et al. (2024), OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments — real OS tasks with execution-based validation; humans >70%, best agents ~12% at publication.
- Zhou et al. (2023), WebArena: A Realistic Web Environment for Building Autonomous Agents — the same gap on web tasks, with reproducible sites.
- Liu et al. (2023), Visual Instruction Tuning — the VLM grounding that computer use depends on.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Extrapolating from the demo. Demos are short, curated tasks; p^N over a dozen real clicks lands somewhere very different, and the benchmarks say so plainly.
- Running it with real credentials on a real machine. It should live in a VM with the narrowest possible permissions, because the failure mode is action, not error.
- Ignoring that the screen is an untrusted input. The agent reads what's on it, so any text an attacker can put on that screen is a prompt injection channel straight into your agent.
At a glance
FieldAI Agents
How it actsscreenshots, mouse, keyboard
Depends onVLM screen grounding
Benchmark realityOSWorld: humans >70%, agents ~12% at publication
DifficultyAdvanced