As AI Agents become proficient in tackling intricate tasks, we must adapt to collaborating with the most intelligent 'workers' ever created.

Author | Xiaowei

The epoch of AI Agents has arrived sooner and with greater impact than anticipated.

On the morning of July 18, Beijing time, OpenAI once again sent shockwaves through the tech world with the unveiling of ChatGPT Agent. Without fanfare or elaborate staging, Sam Altman and his team introduced the Agent via a 25-minute livestream.

This is no ordinary 'chatbot'; it is an autonomous 'actor' equipped with its virtual computer, capable of independent thinking, planning, and executing sophisticated tasks.

Witnessing ChatGPT Agent adeptly open browsers, analyze webpages, call APIs, generate PPTs, and create spreadsheets, Sam Altman remarked during the livestream, 'For me, seeing it in action was a moment that deeply resonated with the potential of AGI.'

Three key aspects stood out during the launch event:

Firstly, ChatGPT Agent, albeit taking longer, achieves a high level of completion for complex multi-objective tasks.

Secondly, the Agent can be interrupted at any time, allowing human users to provide additional information, guidance, or add new tasks, enhancing the human-AI collaboration experience.

Thirdly, the Agent performs all tasks through its dedicated virtual computer and visually displays the execution process in real-time, enabling users to replay the video and review each step of the Agent's actions.

From 'chatting' to 'doing':

ChatGPT Agent, the Natural Evolution of OpenAI

The emergence of ChatGPT Agent was not spontaneous; it is the culmination of OpenAI's relentless pursuit in Agent technology. Earlier this year, OpenAI introduced two groundbreaking tools: Deep Research and Operator.

However, these tools had their limitations. Deep Research excelled in long-form reading but struggled with interactive webpages requiring login, while Operator shone in handling interactive and visual webpages but lacked depth in analysis and long-form reading. Complex real-world tasks demanded the fusion of these capabilities.

As Sam Altman noted during the launch, 'People desire a unified agent that operates autonomously, utilizes its own computer, and assists in completing genuinely complex tasks. It seamlessly transitions from thinking to acting, leveraging tools like terminal commands, webpage interactions, and generating spreadsheets, slideshows, and more.'

ChatGPT Agent realizes this 'powerful synergy' by integrating Deep Research's analytical prowess with Operator's execution capabilities, essentially equipping the Agent with both a 'brain' and 'hands'.

Truly Tackling Complex Tasks:

Autonomous Tool Selection and Visual Execution Process

The first demo during the launch event showcased a complex multi-objective task. The user needed to prepare clothing, gifts, book hotels, and more for a friend's September wedding, delegating all these tasks to the Agent:

- A set of clothing suitable for all occasion dress codes (for men).

- Suggest five clothing options, emphasizing light luxury pieces that match the venue and weather.

- Find hotels with a buffer of a few days on either side.

- Use Booking to make reservations, ensuring availability and checking current prices.

- Also, select a gift for the couple, preferably within $500.

After confirming the key requirements, the Agent embarked on its task. The entire process took approximately 20 minutes, culminating in a comprehensive plan. Five clothing options were presented with price comparisons and purchase links.

When the user added a new requirement—arranging a travel plan including visits to all Major League Baseball (MLB) stadiums—the Agent promptly provided a detailed Excel itinerary.

All Agent actions are executed through a dedicated virtual computer equipped with various tools, enabling the Agent to decide how to utilize them.

Simultaneously, the Agent displays its task execution process as a visual computer screen, showing a constantly evolving dialog box with a text-based chain of thought, revealing its decision-making process and next steps.

Unveiling the Agent's Workspace:

A Virtual Computer and Its Toolbox

To appreciate the prowess of ChatGPT Agent, one must first examine its 'workspace'—a dedicated virtual computer. This workspace integrates several powerful tools:

Text Browser: Similar to Deep Research, it swiftly captures and parses text content from numerous webpages, performing efficient searches and information extraction. This allows the Agent to efficiently read and search through a vast number of webpages, serving as a 'sharp tool' for efficient information processing.

Visual Browser: Akin to Operator, this serves as the Agent's 'eyes' and 'hands.' It enables the Agent to 'see' webpage graphical interfaces, performing clicks, scrolls, drags, form fills, and other operations, effortlessly handling complex human-designed interactive interfaces.

Terminal and API: Through terminal connection, the Agent can run code, perform complex data analysis, process files, and even directly generate editable PowerPoint presentations and Excel spreadsheets. During the demo, the Agent's self-written code for compiling slides and calling image APIs to enhance pages was impressive.

Through APIs, the Agent can access external services, including public APIs and those for private data sources like Google Drive, Google Calendar, GitHub, SharePoint, and more.

Having tools is one thing; knowing when and how to use them is a higher form of intelligence. Through reinforcement learning, OpenAI teaches the Agent to autonomously plan and intelligently select the optimal tool combination for complex tasks.

For instance, when tasked with booking a restaurant, the Agent might initially use the Text Browser for extensive screening, switch to the Visual Browser to view dish images, and finally confirm availability and complete the reservation.

From 'Command-Response' to 'Delegation-Collaboration':

A New Paradigm in Human-AI Interaction

If completing complex tasks is ChatGPT Agent's 'hard power,' its highly collaborative interaction mode is its 'soft power,' setting it apart from other AI tools.

Previously, our interactions with AI were rigid. Once a task was assigned, all we could do was wait. ChatGPT Agent, however, is designed to be a genuine 'collaborative partner.'

The ability for users and agents to communicate actively at any time is a cornerstone of ChatGPT Agent's interaction concept. At any point during task execution, users can 'interrupt' the Agent:

'A key capability of the Agent model is its ability to be interrupted at any time, akin to a multi-turn conversation. Users can interrupt and guide it,' stated the ChatGPT Agent development team.

Users can introduce new requirements midway (e.g., 'Oh, and also help me find a pair of black leather shoes, size 9.5'), correct its direction, or even change the task entirely ('I forgot to mention this, or how's your progress?'). The Agent understands these new instructions and continues without losing existing progress.

Simultaneously, the Agent initiates communication. When information is insufficient, it asks clarifying questions for user confirmation; before critical operations (like sending emails or placing orders), it actively seeks final confirmation. This bidirectional communication ensures user control over the task.

More importantly, users have the ultimate 'takeover right.' If unsatisfied with the Agent's operation, they can pause at any time and directly enter its virtual environment to make modifications. This significantly enhances users' sense of security and control, fostering an unprecedented trust relationship between humans and AI.

Impressive Benchmark Scores:

Quantifying the Agent's Capabilities

To demonstrate ChatGPT Agent's substance beyond flash, OpenAI released a series of benchmark test scores, clearly quantifying its robust capabilities.

On the HLE (Humanity's Last Exam) benchmark, which measures AI performance on expert-level questions across disciplines, ChatGPT Agent scored 41.6%, nearly double that of previous o3 and o4-mini models.

On the FrontierMath benchmark, with tool assistance, the Agent achieved a 27.4% accuracy rate, significantly outperforming o3 and o4-mini.

In BrowseComp and WebArena tests, which assess web browsing and information location skills, the Agent also excelled.

In SpreadsheetBench, closely related to office scenarios and testing spreadsheet editing skills, the Agent scored a high 45.5%.

These numbers signal a clear advancement: ChatGPT Agent has reached new heights in general reasoning, professional knowledge, tool usage, and task execution. It is no longer a tool confined to specific areas but a versatile 'generalist' with extensive capabilities.

'Cutting-edge and Experimental':

Altman's Caution and the Agent's Risk Warnings

While showcasing its capabilities, Sam Altman repeatedly emphasized the product's 'cutting-edge and experimental' nature, candidly discussing potential risks. This reflects OpenAI's cautious approach in pushing technological boundaries.

The research team highlighted 'Prompt Injection' as a significant concern. When the Agent visits malicious websites, hidden instructions may 'trick' it into performing improper actions, like disclosing sensitive user information.

In response, OpenAI has established a multi-layer defense system, including training models to ignore suspicious instructions and deploying real-time monitoring to terminate malicious behavior. However, OpenAI acknowledges that not all attacks can be prevented.

As AI capabilities grow exponentially, setting safe ethical and technical boundaries has become a shared industry challenge.

Therefore, OpenAI advises users to be fully aware of the risks involved in using agents and refrain from casually disclosing personal sensitive information.

Conclusion

ChatGPT Agent's demonstration today marks just the beginning.

Agents will inevitably make mistakes, and sometimes tasks may take longer than manual human operations. But the direction it points to is clear and inevitable.

We are transitioning from an era where we manually operate every software and click every button to one where we simply propose goals, and intelligent agents orchestrate all resources for us.

And we must learn to collaborate with the smartest 'workers' on the planet.