SE Issues Stack Up With More HBM Layers

[admin]

Key Takeaways:

• HBM scaling is running into physical limitations.
• More bandwidth translates into higher area and capacity costs.
• AI demand is intensifying the DRAM shortage.

DRAM is under stress, and the fallout is affecting the entire chip industry. Pressed by large systems companies to move data in and out of memory faster, the big three DRAM vendors — Micron, Samsung, and SK hynix — are wrestling with some increasingly complex tradeoffs. The best solution so far has been to stack more layers in an HBM module, or cube. But each additional layer piles on new challenges, some of which have yet to be solved. Adding more layers requires thinning of memory dies in the stack in order to stay within the HBM package height limit. But thinning DRAM dies is reaching the breaking point — sometimes literally. They are harder to handle during manufacturing and packaging. And the higher the stack, the greater the power density, and the longer the thermal path through the assembled module. Thinner dies also are more prone to warpage, which makes it more difficult to handle wafers, align bumps, and bond dies together. Compounding those problems are constant demands for more bandwidth to handle the explosion in data, particularly in AI data centers. Through-silicon vias (TSVs) are the main data paths 3D-IC modules, and they are essential to support HBM’s very wide parallel interface. The more TSVs, the greater the bandwidth, which is fundamental for improving the performance of any HBM module. But TSVs also consume area on a die, which limits the amount of data that can actually be stored on each layer. And if the size of the HBM module grows vertically on the z axis, it makes it harder to package and harder to dissipate heat in that package, which begins in the logic die at the base of the stack. Collectively, these and other issues are having a significant impact on yield, and therefore on DRAM supplies, according to industry insiders. The more DRAM dies in an HBM stack, the more that can go wrong. In addition to more TSVs and thinner dies, there are voids in underfill, and other structural problems caused by misalignment or process variation. Any one of these can turn the entire stack of dies into very expensive scrap. And while HBM fundamentally still uses the same commoditized DRAM cell technology as any other type of DRAM, HBM consumes significantly more silicon and wafer capacity per bit than conventional DRAM. In his keynote speech at Hot Chips 2026, Jim Handy, general director at Objective Analysis, identified three reasons for DRAM shortages:

• Explosive AI demand and unprecedented spending are increasing demand for extremely fast DRAM;
• Reduced availability of DRAM wafers because HBM requires more than other flavors of DRAM, and
• Insufficient manufacturing capacity to keep pace with insatiable demand.

“You’ve got this AI spending, but then you need HBM to be able to get the kind of bandwidth you want because everybody would like to have an infinitely large cache memory on their processor chip,” Handy said. “They can’t have that, so they say, ‘Okay, how about if we have a huge SRAM? That’s way too expensive. And so they use HBM. HBM has really wide buses, and so it gets a lot of bandwidth in there, even though there is a little bit of latency, because the latency is about the same as standard DRAM.” That bandwidth comes at the expense of area, however. “All these dies are connected through tens of thousands of TSVs,” said Sangwook Han, design team member in Samsung’s Memory Business. “While maximum capacity has increased successfully in each generation, the primary motivation remains bandwidth because it has essentially doubled with each generation. To achieve higher bandwidth, we face two major bottlenecks — TSVs between dies and the PHY I/O in the base die. TSV scaling is relatively straightforward, since increasing the speed of the TSVs is relatively difficult and sometimes risky. So we have been increasing the number of TSVs. The drawback of this approach is the silicon area taken by the TSV array, which forces us to continuously reduce the pitch of the TSVs. But compared to TSVs, scaling of the PHY I/O is much more complicated.”
Fig. 1: Bottlenecks in HBM modules. Source: Samsung Memory/Hot Chips 2026 The amount of area taken up by TSVs is significant. “If you look at the number of gigabytes you get per wafer, it’s a third as much as what you get with DDR — standard DRAM,” Handy said. “What that means is, all of a sudden, HBM just is demanding a colossal number of wafers, and that’s causing all DRAM to go into shortage because they’re all made on the same product process lines.” What’s unique about HBM versus other types of DRAM is that the TSVs and peripheral circuitry consume silicon area that would otherwise be used for memory cells. “The difference is in what kind of technology, packaging, and architecture is designed around that cell technology to be able to deliver higher bandwidth,” said Raghu Sreeramaneni, fellow for HBM Design Architecture at Micron. “The main thing is extremely high parallelism. There are way more banks that can operate in parallel. There are way more data paths that can take that data and connect it to the base die. HBM3E had 128 banks on every DRAM die. HBM4 goes all the way to 256 banks per DRAM die, and then, obviously, all the DRAM dies connect to the base die, which forms the interface to the GPU or XPU. There are essentially two PHYs. The base die has a PHY that talks to the XPU through the microbumps and interposer, and there’s a TSV PHY that is connecting all the DRAMs to the base die.”
Fig. 2: System-in-package diagram showing HBM (upper right) and its connection to a GPU and interposer. Increasing the height of the DRAM stack would make it higher than the GPU, which would make it more difficult to manufacture. Source: Micron/Hot Chips 2026 Handy explained that what caught the DRAM manufacturers by surprise is that they had gotten to a point where the DRAM market was growing slowly enough that they could match the rate of growth in gigabytes just by increasing the number of gigabytes per wafer. “Despite all the talk about Moore’s Law slowing, you ended up having the gigabytes per wafer going up at a satisfactory rate to match that,” he said. “So nobody was adding [new fab] capacity. They hadn’t added capacity for over 10 years, and all of a sudden they’re having to add factories.” There are no quick fixes here. On the capacity side, it takes at least a couple years to build a new DRAM fab. The time frame can vary greatly by region, based upon the availability of power, water, environmental regulations, and the availability of labor to build the fabs. And it’s not at all clear whether the current construction will be sufficient to meet demand and bring DRAM prices back in line. Most experts believe demand for memory is accelerating faster than for compute. “Compute is scaling at roughly 3X every 2 years,” said Sreeramaneni. “HBM, while it has solved a lot of the bandwidth issues and scaled up bandwidth, is lagging and is more like 2X every 2 years. So the memory wall is still present, and in fact, it may be getting worse.”
Fig. 3: Compute demand far outstrips DRAM supply and capability, forcing changes in where data is processed and how much needs to be stored. Source: Micron/Hot Chips 2026 Complicating factors
Just adding more dies into the stack of DRAM chips only complicates matters. SK hynix’s work on 16-high HBM exposed a number of critical manufacturing factors. “There is discussion about increasing the total cube thickness from 720 to 775 microns,” said Jaesik Lee, vice president of package engineering at SK hynix America, in his Hot Chips 2026 keynote. “This helps a lot because we can have more margin from that increased thickness, but we still need to reduce the thickness by 10% compared to the 12-die height. We also can reduce the gap height 50%, and we can do a bump pitch reduction, but that’s separate from the thickness. But this gives us new challenges. One is the die wall pitch, because the die wall gets thinner. We also have a power density increase, and a bandwidth increase. This creates a thermal challenge, especially on the packaging side, because we have more layers of the die that we need to stack, and then we have more oxide layers because each die has oxide layers.” Moving heat up through the HBM module is a growing challenge, too. “If you think of an HBM cube, the heat is getting extracted at the top,” Sreeramaneni said. “The base die is typically the hottest part of the die, because it’s doing some of the most complex work. It has the highest-speed interconnections, so you’re generating a lot of heat at the bottom. Your heat sink and cooling is at the very top. There is thermal resistance through the cube, and DRAM doesn’t like to be hot. So refreshes of DRAM start to become a problem for reliability if it gets too hot. Figuring out what the thermal solutions are is very, very critical, and it’s getting to the point where we are now architecting solutions around thermals instead of the other way around.” Changes ahead
“We can keep making progress on HBM cubes and memory solutions that can deliver higher bandwidths, but at some point that data needs to go from the HBM to the processor, and obviously back and forth,” said Micron’s Sreeramaneni. “There are extremely interconnected dependencies, from process technology to circuit innovations and interposer technology. We really need some new ways of doing high-speed I/O design. We are looking at more memory-optimized SerDes PHYs, instead of more native memory interfaces. Are there things from the typical D2D SerDes locations that can be optimized for memory? Optics is eventually going to come into this space, as well. And given the form factor, and how these all fit together, the interposers themselves are making rapid progress, whether it’s being able to fit bigger SiPs (systems in package), more GPUs, more HBM being integrated together, different materials, better bandwidths in the channels to support higher speeds. So there is a lot of innovation around the high-speed links.” Bonding technology will evolve, as well. “Today, most of the HBM solutions we have are thermal-compression bonded or microbumps,” Sreeramaneni said. “At some point there is a transition to move into more fusion bonding and hybrid bonding, which can support extremely tight pitches with single-digit micron resolution, and which can move much more data through the package. It is also good for thermals, because there are less dielectric layers between each of the dies in the stack, so the thermal resistance is also improved.” All of this will be necessary as AI adoption continues to grow, affecting everything from the AI data center to AI embedded into edge devices. AI has dramatically increased the amount of data that needs to be processed, and the volume will increase as more synthetic data is created and added into the mix. The solution will likely be more distributed processing of that data, including in-memory and in-sensor processing at the edge, and less data being sent from the edge to AI data centers for generative and agentic AI. Conclusion
HBM scaling is evolving from a memory density problem to a system-level one. As more dies are added into the stack, bandwidth requirements will grow. That, in turn, will create challenges involving die thickness, thermal management, and package complexity, and it almost certainly will reduce yield. Future solutions will involve new technologies and approaches, and likely some fundamental changes to how and where data is stored, processed, and reduced, especially with the rollout of edge AI and more distributed inferencing architectures using smaller language models. But there is much work to be done, and DRAM technology is suddenly attracting much more interest than it has for decades. The post Issues Stack Up With More HBM Layers appeared first on Semiconductor Engineering.

Source: https://semiengineering.com/issues-stac ... bm-layers/