Claude’s computer operation capability is now live: how to use it from understanding screenshots to auto-filling forms

Recently, Claude has taken a big step forward on the “can see the screen and click the mouse” front: instead of only answering questions, it now tries to directly operate the computer interface to complete tasks. This article takes a more hands-on angle to clearly explain what Claude’s computer operation capability is, what it’s good for, and what pitfalls to watch out for in real-world deployment.

What exactly is new in Claude’s computer operation capability?

According to public reports, Anthropic provided Claude 3.5 Sonnet with an API approach that enables the model to “perceive the computer interface and interact with it”: Claude can read screenshots, infer the current UI state, then break a goal into a sequence of actions and execute them.

You can think of it as a combination of “image understanding + multi-step operations”: Claude first understands what windows, buttons, and tables are in the screenshot, then decides where to click next, what to type, and how to navigate between pages.

What kinds of work are suitable to hand off to Claude to do directly?

The best fit is computer workflows that have clear rules, repetitive steps, but are time-consuming for humans—for example: opening a browser to search for information, organizing the results into a spreadsheet, and entering data into a back-office system field by field.

When you need “don’t just give me the answer—also run the whole process for me,” Claude’s value shows up: it can plan, execute, and then correct within the same task context, rather than making you copy and paste across multiple tools.

Getting started: make Claude steady before making it fast

In deployment, it’s recommended to design Claude as an “execution-style assistant”: first give clear goals and boundaries (which pages it’s allowed to access, which fields it can modify), then have Claude output a step-by-step plan, and require a second confirmation at critical steps.

If your task involves filling out forms or navigating pages, you can first have Claude use screenshots to produce a “UI element checklist.” After confirming its recognition is correct, move into execution—this makes it less likely to go off track.

Known limitations and pitfall-avoidance tips

Anthropic also acknowledges that Claude’s computer operation capability isn’t perfect. Actions that feel natural to humans—scrolling, dragging, zooming—are still challenging for Claude; in experiments, there were even cases where it mistakenly stopped screen recording, causing content loss.

On the evaluation side, reports mention Claude scored about 14.9% on OSWorld’s screenshot-understanding tasks (rising to 22% when the step limit is increased), still far from human level. So a more pragmatic approach is to let Claude handle workflows that are “reversible and verifiable,” and to add auditing and permission controls for critical actions.