Hello, everyone!

It's been quite some time since I last penned a technical article. With a bit of free time lately, I'd like to delve into how GPUs collaborate and distribute tasks in AI supercomputing clusters.

Let's take Meta's Prometheus data center as a case study to explore how 1 million GPUs operate in harmony within this facility.

Alright, let's dive in officially.

I. Uninterrupted Operation: The Primary Challenge for AI Superclusters

First, let's acquaint ourselves with the Meta Prometheus cluster!

Prometheus is a gigawatt-scale AI supercomputing cluster constructed by Meta in New Albany, Ohio. The initial phase of the cluster is now complete and is being utilized to train the Muse series of foundational models (including the Muse Wonder model). Meta has stated that the site is expanding to accommodate approximately 1 million GPUs.

Envision 1 million GPUs working in tandem—the complexity is mind-boggling. It's akin to enlisting 1 million workers to solve a math problem simultaneously, requiring them to synchronize and exchange computation results every second. However, if even one worker nods off, makes a calculation error, or mishandles a note, the entire project would need to be restarted. This is the daily challenge faced by the Prometheus supercomputing cluster.

Therefore, merely stacking hardware is just the tip of the iceberg—the real technical hurdle lies in ensuring these 1 million 'delicate' chips work in unison 24/7.

II. Packing 72 GPUs into a 'Super Brain' for Enhanced Computing Power

Traditional servers resemble individual rooms, each housing 8 GPUs connected via network cables. However, modern AI models are too vast for just 8 GPUs, and the 'network cables' for data transmission become a bottleneck.

The latest solution is the NVL72 architecture, which crams 72 GPUs into a 'super brain.' These GPUs are no longer connected via external network cables but through a 'high-speed neural bundle' known as NVLink, achieving a bandwidth of 130 TB/s—equivalent to transmitting 30,000 high-definition movies per second. In this setup, the 72 GPUs function together as a single massive chip.

Currently, Meta's cluster is running two generations of these 'super brains':

The first generation, dubbed Catalina, utilizes GB200 chips, each equipped with 192 GB of high-speed memory (HBM3e). Two racks form a group, consuming up to 120 kilowatts of power (equivalent to 120 home air conditioners running simultaneously). Each GPU is paired with a Grace CPU and two ConnectX-7 400G network cards to ensure seamless data flow.

The second generation, named Clemente, upgrades to GB300 chips, increasing memory to 288 GB and boosting performance by 10%. Moreover, it fits 72 GPUs into a single rack, adopting the newer ConnectX-8 SuperNIC network card with native support for 800G high-speed networks.

Next year, the third generation, San Miguel, will be introduced, featuring even newer Vera Rubin chips. A single rack will consume over 170 kilowatts, with memory bandwidth and computing power doubling again.

However, packing so many chips together has a drawback: if one chip fails, it impacts not just 8 GPUs but the entire 'super brain' of 72. It's like putting all your eggs in one basket—the larger the basket, the more catastrophic the loss if it topples.

III. For Stability, Only 64 Out of Every 72 GPUs Are Active

Faced with this challenge, engineers devised a counterintuitive strategy: intentionally allowing 8 GPUs to 'slack off.'

In each 'super brain' of 72 GPUs, only 64 actually handle computation tasks, while the remaining 8 serve as 'reserves' ready to step in at a moment's notice. If a GPU fails, a reserve immediately takes over, allowing training to resume from the nearest 'checkpoint' without any downtime for repairs. This design, known as the '64/72 slicing strategy,' may seem to waste 11% of computing power but significantly reduces downtime.

For extreme cases where an entire rack fails, Meta has prepared an additional 2% of spare racks. To launch a training task requiring 32,768 GPUs, 523 racks (totaling 37,656 GPUs) would actually be needed, with the extra 13% serving as a 'safety net.'

IV. Monitoring 1 Million GPUs Like an 'ECG'

Even with a reserve mechanism in place, failures still occur.

The trickiest issue is signal problems: an NVL72 'super brain' contains 5,184 high-speed signal lines, each transmitting data at 200 Gbps. Even if just one line has poor contact, the entire system will frequently 'stutter.'

To address this, engineers equipped the cluster with an 'ECG monitor'—continuously collecting signal quality data from every link. If a link shows signs of 'arrhythmia' (signal degradation), it is proactively replaced before it fails completely. The chip's firmware (built-in software) can also be updated like a phone system update, pushed to 100,000 GPUs within hours to quickly fix known bugs.

The results are astonishing: in mid-2025, the cluster experienced nearly 1,000 failures per day for every 8,000 GPUs; a year later, the failure rate dropped by 99%. Now, interruptions occur only once every 100,000 GPU-hours, and training efficiency has improved from 85% to over 90%.

V. An 'Information Highway' Spanning Buildings

It's impossible to cram 1 million GPUs into a single room—they are distributed across multiple buildings. This introduces another physical challenge: fiber-optic speed is limited. Electrical signals traveling through fiber optics take 5 microseconds per kilometer, a non-negligible delay for AI training that requires frequent synchronization.

Meta's solution is a 'hierarchical transportation system':

High-Speed Within Buildings: Each building houses about 20,000 GPUs, using short-distance fiber optics to ensure extremely low latency for tasks requiring frequent synchronization.

Inter-Building Trunk Lines: Buildings are connected via a dedicated 'Backend Aggregation' network using 800G-ZR coherent optical modules, spanning distances from 2 km to 100 km, transmitting data that is not sensitive to latency.

This is akin to urban planning: subways (high-speed, low-latency) for intra-city travel and high-speed trains (high-capacity but slightly slower) for inter-city travel. The entire campus is laid out with millions of miles of fiber optics, utilizing tens of thousands of switches and millions of optical modules.

VI. Taming the 'Power Beast'

The most perilous moments often occur during 'start' and 'stop' instants. When tens of thousands of GPUs switch from idle to full-speed computation simultaneously, power consumption can spike by 30%-40% in seconds—equivalent to an instantaneous increase of tens of megawatts in power demand, enough to overload and trip the grid.

To mitigate these fluctuations, Meta developed a 'power throttle' system called Dynamo Dimmer. When the cluster approaches its power supply limit, the system automatically reduces GPU power, akin to easing off the gas pedal while driving. Meanwhile, each rack is equipped with battery packs that act like 'power sponges' to absorb instantaneous power spikes, while the entire campus has a large-scale energy storage station as a backup.

Cooling is also a major concern. Older buildings employ a hybrid 'air + liquid cooling' mode, while newer buildings adopt full liquid cooling—submerging servers directly in cooling fluid. Meta even open-sourced their liquid-cooled rack design for the entire industry to reference.

VII. Next Generation: Designing the 'Perfect Cluster' from the Ground Up

The success of Prometheus is being applied to Meta's next-generation cluster, Hyperion. Located in Richland Parish, Louisiana, this project will be designed from scratch, featuring full liquid cooling, deep coupling with the power grid, and an optimized network architecture for million-scale GPU deployments.

From Prometheus's experience, it's evident that future AI competition is not just about the number of chips but also about systems engineering capabilities. When GPU counts reach the million level, the real technical barrier lies in designing a massive system that can diagnose itself, repair itself, and avoid overloading the power grid.

Behind these 1 million GPUs lies a sophisticated engineering philosophy—not just about making machines run faster but about keeping them running indefinitely.