智能体真的会用电脑吗？a16z 用数据给出答案

Key Highlights

a16z answered "can agents really use computers?" with benchmark data. The best score of computer-use agents on OSWorld-Verified rose from 42% a year ago to 85%. That already surpasses the roughly 72% level of human testers, with Claude Fable 5 leading at 85%. In other words, agents operating graphical interfaces are moving from demo gimmick to deliverable capability, and for the first time they match humans in controlled settings where the task set is fixed and measurable.

What Happened

a16z used data to answer "do agents really use computers?" OSWorld-Verified is a benchmark simulating real desktop environments, asking agents to click buttons, fill forms, switch windows, and run commands like a human to complete multi-step tasks. A year ago the best score was only 42%, barely functional; now it hits 85%, meaning most daily operations can be done autonomously, and for the first time steadily exceeds the ~72% pass rate of human testers. This shows machine operation already has practical value in controlled settings, not just lab tricks performed by researchers tuning prompts by hand for a highlight reel.

Technical Details

The score jump comes from several technical lines converging: first, more stable visual grounding, where models locate interface-element coordinates more accurately instead of clicking roughly in the right area; second, long-horizon planning and retry mechanisms that self-correct on pop-ups or errors rather than giving up; third, running the "observe-think-act" loop longer without losing context, so the agent remembers earlier steps while working toward a goal. Claude Fable 5 leads at 85%, showing that on mapping language reasoning to screen actions, a strong model plus dedicated post-training works markedly, narrowing the gap between "understanding" and "doing it right," and making end-to-end tasks more reliable than before.

Versus Competitors

Horizontally, beyond OSWorld, WebVoyager and AndroidWorld show similar curves: top models rapidly approach humans at "operating graphical interfaces." Compared with pure chat agents, computer-use agents' value is direct takeover of legacy software without needing an open API, thus compatible with a huge installed base of existing tools that were never designed for automation. But note 85% is a controlled-benchmark score; real desktops are full of unseen layouts and permission pop-ups, so generalization still discounts and cannot equal production-environment performance — human-in-the-loop remains needed at deployment to catch the cases the benchmark did not include.

Industry Impact and Use Cases

Put simply, agents "knowing how to click a computer" is no longer a gimmick but a deliverable capability. For RPA vendors, this is a frontal hit — model-based computer-use agents are more flexible and maintainable than hard-coded scripts, and can adapt to UI changes without a full rewrite. For enterprises, it means repetitive office work like reimbursement, data entry, and cross-system reconciliation can be automated by pointing a model at the same screens humans use. But before deployment, two accounts must be settled: fault tolerance for fragile processes, and operation logging with permission control, to avoid agents deleting or overreaching unattended, keeping the efficiency dividend within a controllable range that finance and security can both sign off on. The business read-through is that computer-use agents are about to collide with the entire robotic-process-automation industry, which has sold brittle, selector-based scripts for two decades. Model-driven agents that tolerate UI changes threaten to undercut that model overnight, because the buyer no longer pays a consultant to re-map every button after a redesign. The near-term winners will be teams that wire these agents into human-reviewed workflows: a bot drafts the steps, a person confirms the risky ones, and exceptions route to a queue. Full autonomy is tempting but premature for anything touching money or customer records, so the pragmatic path is assisted operation with strong logs. There is also a labor dimension worth naming plainly: tasks like reconciling two systems or keying in forms are exactly the entry-level office work that absorbs large headcount, and automation there will reshape those roles faster than headline-coding assistants will touch engineering. Companies that pilot now, with guardrails, will learn the failure modes before competitors are forced to by cost pressure, and that head start matters more than the benchmark number. The benchmark is a milestone, but the real test is whether these agents hold up on the messy desktops users actually own.