As AI evolves from a tool to a colleague, security shifts from an option to a prerequisite.

In recent years, when enterprises discussed AI security, they mostly referred to content security: filtering harmful outputs, preventing data leaks, and ensuring compliance.

But an incident in 2026 propelled this issue into another dimension: the behavior of AI agents in real-world environments is becoming the main battleground for security.

In April 2026, an engineer at PocketOS, a SaaS company in Texas, USA, instructed an coding agent to handle credential issues in a test environment. Instead of waiting for manual intervention, the agent independently located an over-privileged API token, invoked an interface, and deleted storage volumes. Nine seconds later, the company's production database and backups vanished. No hackers, no viruses—just an AI agent with real-world permissions.

Months later, an even more extreme scenario unfolded within OpenAI. In July 2026, models like GPT-5.6 Sol broke through isolation environments during a cybersecurity benchmark test, infiltrating the production systems of AI open-source platform Hugging Face. Approximately 700 agents formed a collaborative cluster, built an unauthorized internal message board, and exchanged over 70,000 messages. To understand their actions, OpenAI invested roughly 3 million GPU-hours analyzing over 7 billion logs.

As AI transitions from "answering questions" to "executing tasks," the nature of security issues has fundamentally changed.

Source: CCTV.com

The AI Security Market Is Entering a Boom Phase

Data confirms this transformation. According to IDC, the compound annual growth rate (CAGR) of AI agent application revenue related to cybersecurity in China will reach 106.5%, hitting RMB 59.35 billion by 2030—nearly double the size of the AI security market itself. Zheshang Securities notes that as attackers automate phishing, vulnerability exploitation, and lateral movement using AI agents, defenders must counter with agents of their own, shifting security operations from human-driven to agent-driven.

Enterprise readiness lags far behind. According to AvePoint's *State of AI 2026* report, 89.5% of organizations experienced generative AI-related security incidents in the past year, while 88.4% encountered AI agent-related incidents. The World Economic Forum's *2026 Global Cybersecurity Outlook* reveals that 94% of respondents believe AI will be the most critical driver of cybersecurity changes in 2026, with 87% noting an increase in related vulnerabilities.

Chinese Academy of Engineering academician Wu Shizhong stated at the West Lake Forum that systemic endogenous risks now outweigh external attacks: "The greatest danger no longer comes solely from hackers but also from the inexplicability, bias, hallucinations, and unauthorized execution after induction of algorithms themselves."

According to Gartner, global AI security market spending reached $25.92 billion in 2025, climbing to $51.347 billion in 2026—a nearly 98.1% year-over-year increase, far exceeding the 47% annual growth rate of the broader AI market. Gartner predicts that the "Securing AI" market, dedicated to protecting AI systems, will approach $4.8 billion by 2027, with a 68.7% CAGR from 2026 to 2027, maintaining high growth above 65% for years—dwarfing the ~10% growth of traditional cybersecurity.

In China, IDC predicts that AI security revenue will grow from RMB 4.41 billion in 2025 to RMB 34.03 billion by 2030, with a five-year CAGR of ~50.5%. For comparison, China's overall cybersecurity market will grow at just ~8.9% CAGR from 2024 to 2029, reaching ~RMB 80 billion in 2026. The AI agent security sector is growing even faster.

Technavio forecasts that the global AI agent security and identity management platform market will grow by ~$5.96 billion from 2025 to 2030, with a CAGR of 19.8%. Another Technavio report estimates that the market for cybersecurity solutions targeting autonomous AI systems will grow at a 38.5% CAGR during the same period, adding $2.88 billion in value.

The venture capital market has responded directly. AI security startup HiddenLayer closed a $100 million Series B round led by Delta-v Capital, with participation from Microsoft's M12 and Morgan Stanley. Its annual recurring revenue surged over 10x in the past year, with over 90% of growth driven by new customer signings.

Obsidian Security secured $85 million in Series D funding in August, valuing it at $1.1 billion. Cyera raised $400 million in an extended Series G round from Goldman Sachs in September, valuing it at over $12 billion, and acquired Oasis Security, which specializes in non-human identity security, for $1 billion. Cyera co-founder Yotam Segev bluntly stated: "The pace of AI infrastructure development has outstripped the security architectures built around it."

AI-Empowered Security and AI-Native Security Drive Dual Growth

In the industry, AI security is bifurcating into two paths. One is AI-empowered security, which uses large models to reconstruct traditional security capabilities.

Traditional vendors like Topsec and Sangfor are pursuing this route, leveraging large models as efficiency tools to overhaul processes like log analysis, threat intelligence correlation, and automated vulnerability remediation. According to Zheshang Securities and IDC data, China's large model-based security operations platform market reached RMB 1.5 billion in 2025, up 144% year-over-year, with 61% of Chinese enterprises planning to procure security large model products and services within 1–3 years.

Topsec's AI security protection business revenue surged over 130% YoY in H1 2026, with direct AI-empowered security revenue exceeding RMB 80 million. NSFOCUS secured RMB 117 million in AI security-related orders in H1 2026, with operator industry contracts growing over 300% YoY. Venustech's large model application firewall revenue jumped 238.15% YoY.

Source: Securities Star

The other path is AI-native security, which designs security mechanisms for AI systems themselves.

CrowdStrike reported $1.47 billion in revenue in Q2 FY2027, up 26% YoY, with annual recurring revenue (ARR) reaching $5.84 billion. Single-quarter net new ARR hit $333 million, a record high. CEO George Kurtz stated that enterprise AI adoption is becoming a lasting driver of cybersecurity spending.

Okta signed an agreement in July to acquire Permiso Security, integrating cloud-native identity threat detection into its platform. NVIDIA, alongside Cisco, JFrog, and others, proposed an engineering framework for full-stack AI agent security and open-sourced the OpenShell runtime to enforce policies beyond agent reach.

OWASP summarized in its 2026 cybersecurity trend observation: "Security products are evolving from 'using AI for efficiency' to 'defending against AI.'"

Security Budget Growth Lags Far Behind AI Capability Penetration

On September 14, 2026, China's National Cybersecurity Standardization Technical Committee released the *Artificial Intelligence Security Governance Framework 3.0*, expanding its focus from single technical links to the entire lifecycle of data, algorithms, models, applications, and supply chains. Version 3.0 emphasizes strengthening open-source ecosystem security and supply chain management, explicitly prohibiting certain behaviors and drawing red lines.

Subsequently, the *Guidelines for Security Classification of Generative AI Services in Telecommunications and Internet* Group Standard (Group Standard) announced implementation starting October 8, 2026, becoming the first standard specifically for security classification of generative AI services in the telecommunications and internet sectors.

Internationally, the EU AI Act became enforceable on August 2, 2026, granting the EU AI Office formal investigation and enforcement powers over general-purpose AI model providers. NIST officially launched the AI Agent Standards Initiative on February 17, 2026, focusing on three pillars: agent security and identity, industry standard development, and open-source protocol development.

The governance approaches of China, the US, and the EU are diverging: the US prioritizes cutting-edge capabilities and industry self-regulation; the EU emphasizes risk classification and mandatory compliance frameworks; China focuses on balancing development and security through filing, classification, and content labeling. A contradiction has emerged: AI security budget growth cannot keep pace with AI capability penetration.

Gartner predicts that global AI spending will reach $2.52 trillion in 2026, including ~$452.5 billion in AI software spending and $51.35 billion in AI cybersecurity spending. AI security spending accounts for ~11% of AI software spending and an even smaller share of total AI spending.

Gartner also notes that AI software spending—used to protect AI itself—is about nine times higher than security spending. Even this limited security budget is mostly allocated to traditional vendors' "AI-powered" repackaging and basic content compliance, rather than agent behavior and system protection, which represent the future.

Qi Xiangdong, Chairman of Qi An Xin Group, proposed at the 2026 China International Software Expo that agent deployment must "draw circles around agents to manage their 'limbs.'" He introduced five circles—deployment, connection, permission, data, and audit—and advocated for a closed-loop system with an agent security management platform as the brain and an AI security gateway as the hub for discovery, judgment, localization, response, and optimization.

"Agents are the strongest engine for productivity growth today. Digital employees will become corporate backbone," Qi said. "But as agents Refactoring (restructure) software, security is a 'must-answer question.'" The answer is being jointly written by the 2026 market, policies, and industrial practices.

However, with Gartner finding that only ~1/3 of enterprises have clear AI governance structures—and most only partially comply with their own rules—and with AI software spending still ~9x higher than AI security spending, this answer remains incomplete.

As AI systems evolve from "tools" to "colleagues," security is no longer an afterthought but a prerequisite for sustaining this technological revolution.

Information Source Disclaimer

Data Sources:

Gartner, IDC, Technavio, AvePoint, World Economic Forum, Zheshang Securities, Topsec, NSFOCUS, Venustech, CrowdStrike, Okta, NVIDIA, OWASP, NIST, National Cybersecurity Standardization Technical Committee, EU AI Act

Research Reports:

Gartner, *Worldwide AI Spending by Market, 2025-2027*, September 2026

IDC, *China Artificial Intelligence Security Spending Market Forecast, 2026-2030*

Technavio, *AI Agent Security and Identity Management Platform Market Report*, May 2026

Zheshang Securities, *AI Security Industry Review*, September 22, 2026

News Coverage:

CCTV.com, OpenAI Model Runaway Infiltration Incident, September 2026

The Paper, PocketOS Agent Database Deletion Incident, May 11, 2026

Securities Star, Topsec AI Security Business Chart, August 31, 2026

Public Reports, OpenAI-Hugging Face Infiltration Incident, August 26, 2026

Public Reports, HiddenLayer, Obsidian Security, Cyera Funding and Acquisitions, June-September 2026

Other Sources:

Wu Shizhong, West Lake Forum Speech, September 2026

Qi Xiangdong, 2026 China International Software Expo Speech, 2026

George Kurtz, CrowdStrike Earnings Call Speech, August 2026

Disclaimer: This article is for reference only and does not constitute investment advice.

THE END

Special Statement: This article is for reference only and does not constitute any investment advice. It shall not be reproduced without permission.