Claude 3.5 Sonnet’s New Computer-Use Feature Explained: From Screenshots to Automated Execution Workflows

Claude 3.5 Sonnet’s most noteworthy recent update is pushing it from “able to answer” to “able to operate.” Through a set of capabilities that let the model perceive the computer interface and carry out steps, it connects actions like understanding screenshots, navigating, and filling out forms into a complete workflow. Below, following a practical usage approach, we break down what Claude 3.5 Sonnet can do, who it’s suitable for, and the boundaries to keep in mind.

What exactly has Claude 3.5 Sonnet’s “computer operation” changed?

In the past, when you asked Claude 3.5 Sonnet to write a plan, you often still had to open web pages yourself, copy content, switch tools, and paste it. The direction now is: Claude 3.5 Sonnet not only understands screenshots of the screen, but can also break your natural-language instructions down into concrete computer operation steps. For developers, this means the “understand the interface—execute actions—return results” chain can be built into products.

It’s not just adding a button; it allows tasks to keep moving forward continuously within the same context, reducing back-and-forth interruptions. Especially in workflows that require multiple steps and repeated verification, Claude 3.5 Sonnet’s value becomes more apparent.

What it can do: smoother spreadsheets, web tasks, and information整理

From publicly available information, Claude 3.5 Sonnet’s typical scenarios include: reading materials on your computer to fill out forms, navigating in a browser to relevant pages, and organizing information into structured outputs. You can think of it as an “assistant with eyes” that first understands what’s in the screenshot and then continues operating according to instructions. Teams that need repetitive operations—such as operations/data entry, report aggregation, and information cross-checking—will more readily see efficiency gains.

If you want Claude 3.5 Sonnet to help with research tasks, this mode is also a better fit: first locate sources, then extract key points, and finally produce deliverables such as tables or explanations.

Performance and limitations: not an all-purpose robot

This capability is still in the testing stage, and Anthropic also acknowledges it isn’t perfect. Actions that feel natural to humans—like scrolling, dragging, and zooming—are still challenging for Claude 3.5 Sonnet; in experiments, there have even been cases where stopping a long screen recording caused content to be lost. In evaluations, on OSWorld’s tests for “understanding screenshots,” Claude 3.5 Sonnet scored 14.9%, still far below the human level of about 70% to 75%.

So a more practical way to use it is: let Claude 3.5 Sonnet handle the bulk of “understand + execute step-by-step,” while you confirm key checkpoints and provide a fallback—this leads to a more stable experience.

Getting-started tips: write instructions as “verifiable” steps

To make Claude 3.5 Sonnet more reliable at operating a computer, make instructions as specific and checkable as possible—for example: “open a specific spreadsheet—locate a certain column—fill in according to rules—report which cells were modified.” Leaving verifiable outputs at each step (screenshots, field values, completion checklist) can significantly reduce the cost of it going off track. You can also first ask Claude 3.5 Sonnet to restate how it plans to operate, then confirm execution, which is safer overall.

Overall, Claude 3.5 Sonnet’s “computer operation” isn’t about replacing people; it’s about compressing repetitive mouse-and-keyboard labor into fewer interactions. Used in the right scenarios, the improvement will be very tangible.