Claude Autonomous Task Execution: How to Control Your Computer with AI

Anthropic has rolled out a major update to the Claude 3.5 Sonnet model, introducing a new autonomous task execution feature that allows direct computer control. This means Claude is no longer just a conversational assistant—it can "see" the screen and interact with the interface like a human, opening up new possibilities for office automation and programming.

What Changes Does Claude's Autonomous Task Execution Bring

At the core of this feature is Anthropic's specially designed API, which enables Claude to perceive and interact with computer interfaces. Developers simply input instructions, and Claude converts them into concrete computer operations—such as opening a browser, filling out forms, or checking spreadsheets.

According to official data, in the OSWorld benchmark, Claude 3.5 Sonnet achieved a score of 14.9% in understanding screenshots. While this is below the human-level 70-75%, it already surpasses other AI models. When executing more steps, the score can further increase to 22%.

How to Use Claude's Computer Control to Boost Work Efficiency

For everyday users, Claude's computer control capabilities can significantly reduce tedious manual operations. For example, when you need to gather information from multiple data sources, simply tell Claude what you need, and it will automatically open relevant software, find the information, and complete the filling.

Several companies, including Replit, Canva, and DoorDash, have already started testing this feature. Replit even used it to develop a dedicated application inspection tool that automatically evaluates code performance during programming. This autonomous task execution ability makes Claude ideal for handling repetitive, multi-step workflows.

Claude's New Features: Stronger Coding and Multi-Step Task Processing

In addition to computer control, this update also significantly improves Claude's coding capabilities. In the SWE-bench Verified test, Claude 3.5 Sonnet's score jumped from 33.4% to 49%, outperforming all public models, including OpenAI o1-preview.

In the TAU-bench retail domain test, Claude's score rose from 62.6% to 69.2%. Early feedback from companies like GitLab and Cognition indicates that the new model excels in long-term tasks and multi-step software development processes, working stably for hours. For developers and users who need to handle complex tasks efficiently, this Claude upgrade is certainly worth attention.