McKinsey believes that by 2030, inference will consume 60% of demand, with training retreating to 40%. The key variables are not at the GPU level; memory bandwidth, interconnectivity, and software scheduling determine how much hardware potential can be unlocked.

In recent years, the AI community has focused on training. Clustering thousands of GPUs and competing for accelerators have become the norm.

McKinsey released 'Scaling AI Inference,' noting that by 2026, training and inference will each account for 29% of total demand. By 2030, total demand will rise from 120 GW to 278 GW, with inference at 42%, training at 28%, and inference alone accounting for 60% of AI demand. The annualized growth rate for inference is 36%, outpacing the overall AI growth rate of 30%.

Note: McKinsey collaborated with the Global Semiconductor Alliance, interviewing AI architects from major semiconductor companies and hyperscale cloud providers.

Part 1: Computing Power Shifts from GPU Stacking to Cost-per-Unit Analysis

Corporate decision-makers' priorities have shifted. Instead of asking whether AI creates value, they now ask whether the unit service cost is acceptable when creating value.

Companies pay immediately for computing power and infrastructure, while process redesign, data integration, employee adoption, and productivity realization take much longer. An AI deployment can seem very expensive before it becomes operational.

Executives expect costs to rise significantly after autonomous systems take over more work, even though most adoption and supporting infrastructure may not yet be in place. If the costs do not balance, companies will hesitate to scale up inference. Without cost-effectiveness, scaling is just empty talk.

The cost of a single training session can be estimated.

Inference is an ongoing service, constrained by latency, throughput, and unit cost. Deployments often seem expensive not because the model is inadequate but because the unit service cost has not been reduced to an acceptable level. Training is a one-time investment, while inference must continuously justify its costs.

Leading accelerators, high-bandwidth memory, advanced packaging, and power access all command premiums because supply cannot keep up with demand.

Lasting value will accrue to layers that improve overall system economics. Reducing costs, increasing utilization, cutting latency, and boosting output per watt are the metrics that determine where value lies.

Memory manufacturers that alleviate bandwidth bottlenecks, interconnectivity providers that maintain rack consistency, and cooling and power supply companies that enable higher density all secure their positions by improving overall system performance. Scarcity yields short-term gains, while system economics deliver long-term success.

Short-term gains are concentrated among memory manufacturers, leading accelerator companies, and a few custom chip and interconnectivity firms.

Proliferation will wait until constraints shift away from computing power. Power supply, cooling, connectivity, and sensing will create opportunities for differentiation among analog, MCU, and other fabless companies.

Whoever can solve a problem at the system level, rather than just adding another impressive line to a spec sheet, will secure a position. Specialization is valuable only if it addresses system-level issues. Pretty datasheets are not enough.

Part 2: Workload Differentiation and Specialized Chips: The Physical Limits of Computing Power

Today, many existing devices are designed for training but used for inference, creating mismatches beyond just specifications. Training rewards raw computing power, while decoding rewards memory bandwidth. For each token generated, model weights and accumulated KV caches must be moved from memory for minimal arithmetic operations.

Bandwidth, not peak computing power, determines how quickly a node can serve requests. The GPU-HBM combination will continue to handle much of the workload. However, as inference demands become more specific, the single default configuration will loosen, and multiple configurations tailored to bandwidth- or capacity-heavy workloads will coexist.

McKinsey breaks down the 2030 inference token demand by workload.

Text and code for large language models will account for 82%, non-large models 13%, and speech the remainder. Agents, code, conversational search, and similar tasks require 10 billion to 100 billion parameters and contexts up to 128,000 tokens, favoring computation and memory; writing, machine-to-machine communication, media generation, physical AI, and speech all require under 10 billion parameters.

For individual requests, compliance tools reading 300-page contracts are prefilling-heavy and computing-bound, while voice assistants streaming tokens one by one are decoding-heavy and bandwidth-bound. On the same hardware, each pays for unused capabilities. The same card wastes resources on both ends.

The industry is redesigning around this divergence.

Hyperscale cloud providers and AI labs are customizing accelerators for their highest-volume workloads. Groq, Cerebras, SambaNova, and d-Matrix, with their SRAM-based approaches, focus on the bandwidth-bound half. Split configurations are also increasing.

◎ NVIDIA and Groq have signed licensing and talent transfer agreements. AWS uses Trainium for prefilling and Cerebras for decoding.

◎ AMD partners with Cerebras.

◎ Intel and SambaNova have outlined blueprints to assign prefilling to GPUs and decoding to reconfigurable dataflow units.

◎ OpenAI's Jalapeo takes the opposite approach, with a balanced accelerator activating the right combination of computing power, memory, and networking for each phase. Model states remain local without transfer. Chiplet architectures are also being widely explored to enable custom solutions on standard computing platforms.

Specialization has a flaw: chip cycles are measured in years, while model architectures change monthly. By the time silicon is customized for a workload, that workload may have shifted.

Shortening cycles has become an industry goal. Google's TPUs have used reinforcement learning for floor planning across generations. Jeff Dean has publicly stated that AI methods can reduce design cycles from 18-30 months to 3-6 months.

OpenAI's Jalapeo is the strongest evidence yet. Developed with Broadcom, it went from first RTL version to manufacturing tape-out in just 9 months, compared to the 18-24 months required for a custom ASIC from scratch.

OpenAI used its models to explore implementations, compress verification loops, and optimize arithmetic circuits. Machine learning-based floor plan optimization has also become standard in mainstream EDA flows rather than experimental.

Part 3: Open Standards and Sovereignty: Survival Rules Beyond Scale

Accelerators remain central, but they alone cannot determine inference economics. Hardware sets the ceiling; software determines how much of that ceiling is reached.

On-chip performance is determined by three interdependent factors: logic for computation, memory for feeding data, and networking for scaling. As context lengths grow and agents increase KV cache traffic, memory capacity and bandwidth determine how many concurrent requests a node can handle, while networking determines how large a rack a single request can span. Beyond a certain point, latency erodes gains.

Power supply and cooling, including liquid cooling and microfluidics, constrain density before computing power does.

Near-term cost curves may be won by software: caching reuse eliminates redundant prefilling, small models handle simple queries before escalating to complex ones, and batch scheduling optimizes data placement. Cumulative gains make hardware run cheaper.

As AI becomes embedded in critical processes, governments and businesses care more about where data is processed, which model is used, where infrastructure is located, and who can see outputs.

Demand is diverging, not converging. A health ministry, defense department, and regional bank may all need the same capabilities but face constraints on data residency, approved suppliers, model provenance, and local ecosystems.

Sovereignty is not the only driver; ordinary competition matters too. Cloud providers optimize their accelerators, vendors protect full-stack platforms, and robotics and automotive companies design according to their own latency, safety, and power envelopes. All have reasons to differentiate.

The likely outcome is multiple inference stacks coexisting. Sovereignty simply adds another layer of constraints to the divergence.

In this fragmentation, open models and standards are economic necessities, not ideological choices.

◎ Open weights let operators bring the same capability into sovereign clouds, private clusters, or edges, adjusting to locally available silicon.

◎ Closed models offer leading capabilities and integration but lock more of the stack into a single vendor's ecosystem, making it harder for customers to control deployment, data, and underlying layers.

Open interconnectivity and software standards do the same on the hardware side. Specialized architectures need volume to amortize costs. If serving fifteen jurisdictions requires fifteen software stacks, the integration burden will push buyers back to easily replicable general-purpose platforms.

Universal standards let suppliers adapt specialized architectures to local requirements without building separate software for each jurisdiction, preserving the scale that makes specialization viable.

Summary

Infrastructure is reorganizing around the value of each workload, not the capabilities of each component.

No single company or layer can balance inference economics alone, as service costs are determined by silicon, memory, networking, software, power supply, cooling, and data architecture together—and these decisions are hard to reverse once made.