- The push toward 1-megawatt racks is forcing fundamental changes in data center architecture, including cooling, power delivery, rack design, and 3D-IC packaging.
- Higher rack densities may not be the only viable scaling path, as optics, distributed compute, and more specialized silicon could reduce the need for extreme centralized power density.
- AI and agentic workloads are shifting systems from average-power assumptions toward sustained peak-power operation, making power delivery, thermal management, and design validation far more critical.
Higher voltage only addresses one side of the equation. As rack power climbs, the industry also must decide whether ever-denser centralized infrastructure is the most practical scaling path, or whether efficiency gains will come from changing where and how processing is done. Steve Roddy, chief marketing officer at Quadric, said that while advancing the state of the art in data center power distribution and cooling can undoubtedly deliver increased throughput and performance, it is worth asking whether this extreme engineering is the best way to solve a compute challenge. “Our industry has a decades-long history of oscillating between chasing general-purpose compute and then pivoting to more application-specific compute that is less general yet far more energy- and cost-efficient,” Roddy said. “Similarly, the industry over the past 60 years has seen waves of centralized compute (mainframes in the 1960s), followed by waves of distributed compute growth (the PC era of the 1980s and 1990s), followed yet again by the cloud and today’s hyperscalers. If both of those waves intersect in the coming years — more specialized silicon and a pivot back to distributed compute — then the monumental scaling challenge of the AI build-out will see not just continued scaling of generalized GPU compute in data centers, but also more and more special-purpose chips designed for efficient data center inference or training, lessening the power density of chips in the data center with more efficient devices delivering higher inference per watt. And the need to build endless oceans of warehouses filled with racks can be lessened by deploying AI token generation capability into tens of millions of homes and businesses – generation capability that works in tandem with centralized models running on centralized farms of servers.” Optical adds another piece of the alternative puzzle. “As a company, we’re betting against the 1-megawatt rack getting widespread traction,” said Vishal Chandrasekar, director of product management at Ayar Labs. “Why are people going toward those sorts of power? It is because they’re trying to cram more GPUs within a single rack, keep the distance between the GPUs small, and stick with copper. What we’re saying is, ‘Okay, you don’t want to go to optics for whatever reason. Maybe you’re concerned about reliability, cost, or whatever it may be. Instead of extending in the distance direction, you’re expanding in the thermal density direction. We’re trying to go from 200- to 600-kilowatt to 1-megawatt racks. Where do we go from there? Even if you get to 600kW, are you going to go to 1, 2, 3, or 5 megawatts?’ It’s not a realistic scaling vector because every single time you do that, you’re re-architecting the entire data center in terms of how much power it can hold, and sometimes even how much weight the floor can hold. You look at AMD’s Helios racks. I believe it weighs as much as an elephant. So we don’t see that as a scaling vector.” Optics enables the same number of GPUs to be connected within a scale-up domain to 500,000 or even beyond. “In that case, length is released as a vector, and you don’t care about it anymore,” Chandrasekar said. “You can spread out your GPUs across an entire row, because in data centers, people don’t care about real estate costs so much. Power is the real limiting factor. As long as I can spread out my GPUs, I can still stick with, let’s say, 200-kilowatt racks, but have 10 of them connected via optics. I achieve the same performance as getting a one- or two-megawatt rack, and now I have to re-architect my entire data center. I have to re-architect my entire power delivery and power distribution. That’s a far bigger lift, and I have to do that every two years.” Power delivery is a growing challenge
The scaling question becomes even more urgent when viewed from the power-delivery side, where rising rack densities are forcing changes not only in interconnect strategy, but also in the electrical architecture of the data center itself. “The rising data center power needs are driving a huge add-on to data center infrastructure, especially AI data centers that are being built, so we get the news daily that the hyperscalers plan additional data center capacity in the hundreds of megawatts, or like gigawatt scale,” said Christian Hoefling, principal engineer at Infineon Technologies. At the rack level, we talked in the past about maybe below 100-kilowatt power consumption. Today, we are more in the range of 250 kilowatts. In the future, we will talk about up to 1 megawatt in 1 rack. At a GPU level, we’re coming from maybe 100 to 300 amps of peak current in the CPU world. Going over to modern GPUs, we already see today we are in the range of 1,000 amps, 2,000 amps, and we’re expecting these peak currents to hit 10,000 amps per GPU at the end of this decade.” That has big implications for power losses and power distribution. “The power distribution losses in the board itself keep on increasing,” Hoefling said. “Keeping the power delivery resistance the same and doubling the peak current means a factor of four in the power losses, because power losses go with the current squared.” These factors contribute to architectural changes (e.g., 12V to 48/50V to 800V AC, to high‑voltage DC) that are being driven by these rising power needs. “When moving to high‑voltage DC distribution and changing architectures, at the end of the day, it’s all about making it more efficient,” he said. “And an efficiency boost is when you can eliminate [power conversion] steps. Every conversion step that you can eliminate is again a gain in efficiency.” Further, the drive toward higher DC power distribution voltage is simply to meet the requirements that come with higher power in the racks. “If you talk about 1 megawatt of power consumption of a server rack with 800 volts, with a 50-volt busbar, you’re more in the range of like 20,000 amps that you need to support through the busbar,” Hoefling said, noting that data center architectures must change to cut I²R losses. “We see that in the data center and around the data center, the power delivery architectures are changing completely, and this is what requires new technology. And for the end of the decade in gigawatt data centers, we see that there will be a switch from an AC power distribution to a high‑voltage DC power distribution on the complete data center to eliminate some of the power conversion steps, and then to simplify the power delivery to the racks.” Those infrastructure changes are only part of the story. The workload side is shifting, as well, and agentic AI is beginning to reshape power assumptions by turning what used to be intermittent peaks into far more sustained operating conditions. Agentic, long‑running workflows are changing hardware balance and demand, which ties directly into sustained power needs. “With these long‑running agents and all the orchestration they require, the old model of designing for average power and just handling occasional peaks breaks down,” said Sathishkumar Balasubramanian, head of product for EDA AI & Solido at Siemens EDA. “For many of these AI workloads, you have to assume it’s effectively peak power all the time, so power management and analysis become absolutely critical.” AI/agentic workloads are changing the power profile from mostly idle/average‑based design to near-constant peak, forcing much tighter power management and analysis. “Agentic AI creates very irregular, and in some cases truly 24/7 utilization patterns,” Balasubramanian said. “You can’t size the system around an average workload anymore. You have to architect it for sustained peak operation and make sure the entire power and thermal stack can live with that. That means you must make sure that the power management is done very well, and make sure that the power analysis is done and everything is good.” Another added pressure on already extremely complex designs is that AI agents and verification and implementation are more about thermal and 3D‑IC effects that result from the higher switching activity and density driven by AI workloads. Architectures must scale up (more cores, 3D‑IC, etc.), which implicitly raises power density and power‑delivery requirements. “With the pace that development is happening, along with the changes in architectures and new design starts, it’s a good time to be on the verification side,” Balasubramanian said. “Everything is going to be a die stack, which means understanding the physical effect of everything you’re doing. You can have a very high switching bus, but what does it mean in terms of your thermal map?” Agentic AI tends to push designs toward heavier, more continuously engaged compute. “One of the key things in agentic AI is context swapping,” Balasubramanian said. “The biggest thing about the ultimate metric in terms of agentic AI is how fast a processor or system can save, suspend, and restore a complex model because people might be changing back and forth. You don’t want them to wait.” As rack power scales toward 1 megawatt, power increasingly becomes a proxy for compute density. However, just adding more processing elements does not guarantee higher application performance. “The ability of those accelerators to communicate efficiently becomes equally important,” Cadence’s Khan said. “Scale-up interconnects must deliver higher bandwidth, lower latency, improved power efficiency, and broad ecosystem interoperability. AI training and inference workloads are increasingly constrained by memory bandwidth, synchronization overhead, collective communication efficiency, and tail latency. Packing thousands of GPUs into a rack-scale environment only creates value if those GPUs can operate as a coherent computing system. This is where emerging open scale-up standards, such as UALink and other next-generation accelerator interconnect technologies, become increasingly important. They provide the foundation for building large accelerator domains while promoting ecosystem interoperability and innovation.” Shrinking market windows
The AI-driven pressures that are driving new power, cooling, and packaging choices are also compressing market windows, leaving companies less time to redesign, validate, and deploy these architectures before the next generation arrives. Market windows are shrinking, and they will continue to shrink, said Manmeet Walia, executive director, product management at Synopsys. “They only have a year’s market window to take that product into production. If they miss the market window, they might as well kill that generation and move on to the next one. From our perspective, we must be even faster. The specs are ratified within a 1.5- to 2-year window. We need to have a full subsystem test chip available by the time the spec is ratified. For example, in the world of memories, there are three HBM vendors. We need to align our test chip, a subsystem test chip that is going to look like the IP that will be going into customers’ SoCs by the time these devices are made available. Let’s say Samsung has HBM5 ready on this date. We need to have our full test chip so that they can plug it in, build the system, and test it out, bearing in mind that our test chip has to be packaged on a co-op substrate with the HBM memory. And that takes many months, which takes many months off the time you have to get that test chip done.” Advanced packaging extends timelines, as well, creating pressure to prepare earlier so that the entire cycle can happen smoothly. The goal is to have the final test vehicle ready and validated by the next deployment stage, which is non-trivial. “These programs are very expensive, running into a billion dollars for an accelerator, so they cannot go wrong on these,” Walia said. “They cannot take the unverified, unvalidated IP. Being ahead of a standard, which is already going at a pace faster than the silicon cycle, is keeping us up at night. It’s not sustainable long-term. At some point, this whole madness needs to end.” Might that look like more reasonable development? “One of the mitigating things for that is models,” said Synopsys’ Wilkinson. “People talk about models always getting bigger, and the biggest models are getting bigger. But the workhorse models that form 70% of the inference workload are probably midsize — a trillion parameters rather than 10 trillion. So, if you look at something like Google’s suite of consumer AI models, there may be a trillion parameters or even smaller. A lot of that workhorse inferencing, which is where the majority of tokens are being generated, has fewer requirements. There’s an element of focus on TCO being very disciplined about all those workhorse considerations, and then you’ve got the huge training runs and the really big models like Claude and the like, which need a somewhat different infrastructure. Even within that whole ecosystem, a lot of the agentic stuff can be smaller models, and sometimes it is, so it’s not like everything’s massive. There are massive things, but you see with some CSPs, they have more than one platform, and they might use Nvidia or AMD for some things, but they’ve also got their own internal infrastructure that is somewhat differently focused.” This is a layered and complicated picture where a lot of the internal programs may be more directed at the workhorse models that they’re running, as opposed to the flagship models that get everybody’s attention. So maybe everything doesn’t run on a 1-megawatt rack. Wilkinson believes there will be more fragmentation in these solutions, whether they’re internal solutions from big hyperscalers or from startups or other players who are trying to enter the market, maybe on-prem at enterprises. “Not everyone wants to get their AI from the cloud,” he said. “An enterprise is probably not going to put a 3-ton, 500-kilowatt rack on its own premises. They will need solutions that work on-prem within the IT room capabilities they have, and for the workloads they have, which might not even be LLMs. It’s a huge market, and there’s no possibility that any one player can corner all of the workloads and use cases. The way the madness ends is things settling out into segments and specializations with different equipment for different use cases, some of which won’t be on quite such a compressed timescale or bleeding-edge technology.” That fragmentation, however, does not make the integration challenge go away. If anything, it makes coordination across the full stack even more important. Cadence’s Khan noted that the transition to 1-megawatt racks is not fundamentally a rack problem. It’s an ecosystem integration problem. “Success requires coordinated innovation across power delivery, cooling, networking, interconnects, memory systems, optics, software, and data center infrastructure,” he said. “The winners in the AI era will not be the companies that optimize a single component in isolation. They will be the companies that optimize the entire path, from grid to chip, chip to chiller, and accelerator to accelerator, treating the rack, the network, and ultimately the data center as one integrated AI system.” The post The 1-Megawatt Rack Debate appeared first on Semiconductor Engineering.
Source: https://semiengineering.com/the-1-megawatt-rack-debate/