Computer Use — Vision Agents Cost 45x More Than Structured APIs
AI relevance: Vision-based computer-use agents (browser-use, Anthropic's computer use) are increasingly deployed for automating internal tools and web workflows — understanding their cost structure, failure modes, and security implications is critical for anyone running AI agent operations at scale.
What happened
- Reflex published a head-to-head benchmark comparing a vision agent (Claude Sonnet via browser-use 0.12) against a structured API agent (Claude Sonnet with tool-use) on the same admin-panel task.
- The task: find a customer, locate their pending order, accept reviews, and mark delivery — requiring filtering, pagination, cross-entity lookups, and writes.
- The API agent completed it in 8 deterministic tool calls with ±27 token variance across 5 trials.
- The vision agent failed outright on the same prompt — it found only 1 of 4 pending reviews because it couldn't paginate past the visible fold. It had no signal that more content existed.
- With a manually written 14-step UI walkthrough prompt, the vision agent succeeded but consumed 407k–751k input tokens over 14–22 minutes per run — roughly 45x the API agent's cost.
- High variance is inherent to vision agents: wall-clock time spanned 749s to 1257s, and the screenshot-reason-click loop introduced enough non-determinism that a single run isn't a reliable cost estimate.
Why it matters
- Teams deploying computer-use agents are implicitly accepting unpredictable costs and silent failure modes — the vision agent quietly missed 3 of 4 reviews without error.
- The 14-step walkthrough prompt represents hidden engineering overhead — writing and maintaining UI-level instructions for every internal tool negates the "no API needed" value proposition.
- From a security standpoint, vision agents process rendered page content including any injected or dynamically loaded elements — this expands the prompt-injection attack surface compared to structured API calls with defined schemas.
- The benchmark reinforces that MCP servers and structured APIs are the production path for agent automation — computer use remains a stopgap for systems where API surfaces don't exist.
What to do
- When deploying computer-use agents, budget for high token variance — capacity planning based on a single test run will under-estimate costs by 2x or more.
- Implement validation checks on vision agent outputs — the agent's confidence doesn't correlate with completeness (it happily accepted 1 review and moved on).
- Prioritize building structured interfaces (MCP, REST) for internal tools that agents interact with — the cost delta (45x) makes this an economic imperative, not just a technical one.
- Audit vision-agent prompts for page-injection risks — any content the agent sees on-screen can influence its behavior through visual prompt injection.