SA Article Coverage Review · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm
Coverage Summary
- Source: 開啟原始 SA 文章
- Atomic Claims:
35 - Source blocks:
232 - Blocks with ≥1 Atomic Claim:
26 - Blocks without Atomic Claim:
206 - Unplaced Claims:
0
Coverage Review
請從頭到尾閱讀下方 SA 全文。Atomic Claim 會依 Evidence 在原文出現的位置 inline 插入。
若該英文段落已有繁中翻譯 cache，翻譯只會作為淡色閱讀輔助顯示；不會進入 source、Claim provenance 或 Graph。
沒有 Claim callout 的段落不一定有問題；若內容重要且應形成知識，請記到 Missing Claim Notes。
Missing Claim Notes
- 若看到重要但沒有 Atomic Claim 的段落，請在這裡記錄：
- Section:
- Evidence:
- 為什麼重要／應該抽成什麼 Claim：
SA Full Text + Translation + Atomic Claims
Scaling the Memory Wall: The Rise and Roadmap of HBM
The first portion of this report will explain HBM, the manufacturing process, dynamics between vendors, KVCache offload, disaggregated prefill decode, and wide / high-rank EP. The rest of the report will dive deeply into the future of HBM. We will cover the revolutionary change coming to HBM4 with custom base dies for HBM, what various different accelerators are doing with custom HBM including OpenAI, Nvidia, and AMD, the shoreline area problem, memory controller offload, repeater PHYs, LPDDR + HBM combos, and various beachfront expansion techniques. We will also discuss SRAM tags, compute under memory, supply chain implications, and Samsung.
A Brief Overview of HBM
As AI models grow in complexity, AI systems require memory with higher capacity, lower latency, higher bandwidth, and improved energy efficiency. Different forms of memory have different tradeoffs. SRAM is extremely fast but low density. DDR DRAM is high density and cheap but lacks bandwidth. The most popular memory today is on-chip HBM which strikes the balance between capacity and bandwidth.
Source: Rambus
HBM combines vertically stacked DRAM chips with ultra-wide data paths and has the optimal balance of bandwidth, density, and energy consumption for AI workloads. HBM is much more expensive to produce and has a warranted price premium to DDR5, but demand remains strong for HBM. All leading AI accelerators deployed for GenAI training and inference use HBM. The common trend across accelerator roadmaps is to scale memory capacity and bandwidth per chip by adding more stacks, higher layer counts, with faster generations of HBM. Architectures that rely on other forms of memory offer sub-optimal performance, as we have demonstrated ↗.
Atomic Claim 1/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0001
Claim: HBM 以垂直堆疊 DRAM 與 ultra-wide data paths，在 AI workload 中平衡 bandwidth、density 與 energy consumption。
Frame:NARY_RELATION· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 2/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0002
Claim: 目前領先的 GenAI training / inference AI accelerators 都使用 HBM。
Frame:RELATION· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
In this report, we will examine HBM’s present state, what’s happening in the supply chain, and the groundbreaking changes happening in the future. We’ll examine HBM’s critical role in AI accelerator architecture, the impact HBM is having on the DRAM market, and why it is upending the way memory market analysis is being performed. For subscribers, we will also address the major questions on Samsung’s future viability as a supplier, as well as highlight one technological change that may reverse the trend of increasing HBM capacity.
HBM Primer
First, a brief primer on HBM - what makes it special and challenging to manufacture. While HBM is commonly associated with multiple DRAM dies stacked in a 3DIC assembly, the other key feature is HBM’s much wider data bus, improving bandwidth even with mediocre signaling speeds. This significantly wider bus results in HBM being far superior in terms of bandwidth per package than any other form of memory.
Atomic Claim 3/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0003
Claim: HBM 的 wider bus 使其 bandwidth per package 顯著高於其他記憶體形式。
Frame:ATTRIBUTE· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Source: SemiAnalysis
The implication of having much more I/O is increased routing density and complexity. Each I/O requires an individual wire/trace, with additional wiring required for power and control. For a HBM3E stack, there are over a 1,000 wires between the adjacent XPU and the HBM. This level of routing density is not achievable on a PCB or package substrate; therefore, an interposer (silicon or organic) in a 2.5D package assembly like CoWoS is required.
Atomic Claim 4/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0004
Claim: HBM3E 與相鄰 XPU 之間超過 1,000 條 wires，routing density 高到需要 interposer / 2.5D package（如 CoWoS），無法只靠 PCB 或 substrate。
Frame:NARY_RELATION· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
To reduce latency and energy consumption for data transfer, HBM needs to be placed directly adjacent to the shoreline of the compute engine. This makes shoreline (the edge of a SOC) more valuable as HBM can only be limited to the 2 edges of the SOC with the other 2 edges reserved for I/O off the package. This limits the area HBM can be placed in and requires vertical stacking of memory die to provide enough capacity.
Atomic Claim 5/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0005
Claim: 為降低資料傳輸 latency 與 energy consumption，HBM 必須直接鄰接 compute engine shoreline。
Frame:ATTRIBUTE· Mode:ASSERTED· Mapping:PARTIAL
開啟逐條審核
Atomic Claim 6/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0006
Claim: HBM 通常受限於 SoC 的 2 個 edges，另外 2 個 edges 留給 off-package I/O，使 shoreline 成為稀缺資源。
Frame:ATTRIBUTE· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
To enable the 3DIC form factor, each layer of the stack needs to have TSVs (excluding the top of stack) that can deliver power and signal to the layer above. The additional area required to fit these TSVs is what makes HBM die sizes larger than their DDR equivalent: SK Hynix D1z DDR4 has a bit density of 0.296 Gb/mm2, 85% more dense than their HBM3 which is 0.16 Gb/mm2. This TSV process is one of the key differences between standard DRAM, and tooling for this is the main bottleneck when it comes to converting regular DDR DRAM wafer capacity to HBM capacity.
Atomic Claim 7/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0007
Claim: HBM 3DIC stack 除最上層外，每一層都需要 TSV 以向上一層傳遞 power 與 signal。
Frame:RELATION· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 8/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0008
Claim: SK Hynix D1z DDR4 bit density 為 0.296 Gb/mm²，比 HBM3 的 0.16 Gb/mm² 高 85%。
Frame:COMPARISON· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 9/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0009
Claim: TSV tooling 是將 DDR DRAM wafer capacity 轉換為 HBM capacity 的主要 bottleneck。
Frame:RELATION· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
The other difference is at the back-end, where the HBM needs to be stacked to 9 or 13 layers in total (8/12 DRAM layers on top of a logic base die at the bottom). Along with CoWoS, HBM has brought packaging technology to the mainstream. Niche packaging technologies like MR-MUF have now become common knowledge for industry participants.
Atomic Claim 10/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0010
Claim: HBM back-end stack 總層數為 9 或 13 層，即 8/12 層 DRAM 加底部 logic base die。
Frame:ATTRIBUTE· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Explosive Bit Demand
We can see the huge growth in HBM bit demand that has come alongside AI accelerator demand. Despite the rapid rise of custom ASICs, Nvidia will still command the lion’s share of HBM demand in 2027, driven by its aggressive roadmap, where Rubin Ultra alone pushes per GPU capacity to 1 TB. Broadcom follows as TPU and MTIA volumes surge, while incremental OpenAI and SoftBank projects add a smaller but noticeable lift. Amazon also emerges as one of the top HBM customers. For Amazon, it has a strategy of procuring HBM directly rather than through design partners, helping them to lower their cost. For a detailed, chip-by-chip bit forecast, refer to our Accelerator Model ↗. It includes projections for revenue and bit demand by memory vendor, as well as wafer starts and TSV capacity, broken down by supplier. The model also tracks HBM pricing across different generations, and provides chip-by-chip HBM type, layers, stack counts, capacity and bandwidth.
Atomic Claim 11/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0011
Claim: SemiAnalysis 預期 2027 年 Nvidia 仍占 HBM demand 最大份額，Rubin Ultra 單 GPU HBM capacity 將達 1TB。
Frame:ATTRIBUTE· Mode:EXPECTED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 12/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0012
Claim: Amazon 採直接採購 HBM、而非透過 design partners 的策略，以降低成本。
Frame:RELATION· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Source: SemiAnalysis
Process Flow: Front End
When regular DDR DRAM capacity “converts” to HBM capacity – the main change is the addition of tools for forming the TSVs, and more bumping capacity as HBM wafers are bumped on both sides. Both of these steps are to enable the 3D stack, although they are omitted for wafers used for the top die which only need a single side of bumps and don’t require TSVs.
Atomic Claim 13/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0013
Claim: DDR DRAM capacity 轉成 HBM capacity 的主要新增製程是 TSV tooling 與更多 bumping capacity；HBM wafers 需要雙面 bumping。
Frame:NARY_RELATION· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
TSVs require etchers to create the vias, and deposition and plating tools to fill them. To reveal the TSVs, grinders, another etch step, and temporary bonders to attach carrier wafers used in this process are required. This is why HBM capacity is now quoted in terms of TSV capacity, as this is the main incremental set of processes that turn DDR wafers into HBM wafers.
Atomic Claim 14/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0014
Claim: TSV 製程需要 etch 建 via、deposition/plating 填孔，以及 grinding、再次 etch 與 temporary bonding 來 reveal TSV。
Frame:NARY_RELATION· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
For bumping, this is mainly deposition, plating and stripping. Also, optical inspection tools from Camtek and Onto are available to check that the bumps are not defective and are of the correct profile.
Source: Applied Materials
Process Flow: Packaging
The other part of this is back-end packaging which Hynix continues to push forward with MR-MUF which we wrote about extensively here ↗. In short, MR-MUF offers higher productivity and better thermal performance. Hynix’s proprietary (co-developed with NAMICS) molded underfill material offers more thermal dissipation than the Non-Conductive Film that Micron and Samsung use. Hynix has been able to sidestep thermal compression bonding (TCB) because they have found other ways to manage warpage. One of the benefits of TCB is the use of force to stabilize the bonding matter.
Atomic Claim 15/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0015
Claim: MR-MUF 提供較高 productivity 與較佳 thermal performance。
Frame:ATTRIBUTE· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 16/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0016
Claim: Hynix/NAMICS molded underfill 的 thermal dissipation 高於 Micron/Samsung 使用的 NCF。
Frame:COMPARISON· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 17/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0017
Claim: Hynix 因能以其他方式管理 warpage，因此可以避開 TCB。
Frame:RELATION· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
The use of force on the other hand increases the risk of causing damages to bumps. With stress applied to bumps, SK can also add more dummy bumps which also helps with thermal dissipation.
The process is also far more productive. A batch mass reflow and single over-mold step are used for joint formation, compared to a complete TCB step for joint formation for each and every layer with TC-NCF.
Atomic Claim 18/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0018
Claim: MR-MUF 類流程以 batch mass reflow + single over-mold 形成 joints，相較 TC-NCF 每層都需完整 TCB step，productivity 更高。
Frame:COMPARISON· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Source: SK Hynix
Process Flow: Yields
HBM is a more technically sophisticated product than other DRAM form factors, especially given the high 3DIC stack. As such, packaging yields are not going to be comparable to what manufacturers are accustomed to compared to their more conventional products. However, yields at the front end are also challenging, and we think yield is more of a problem at the front end. As mentioned above, HBM is not demanding with regards to speed bins, so why is this the case?
Atomic Claim 19/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0019
Claim: SemiAnalysis 判斷 HBM yield 問題更偏向 front-end，而非只有 packaging。
Frame:ATTRIBUTE· Mode:INFERRED· Mapping:PARTIAL
開啟逐條審核
The reason goes back to the 3DIC assembly and the TSVs. One of the challenges is the Power Distribution Network (PDN) with the TSVs needing to be able to deliver power up the stack. TSV layout and design is proprietary and one of the main areas of differentiation between the various manufacturers.
Atomic Claim 20/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0020
Claim: TSV layout/design 為 proprietary，且是 HBM 廠商間主要 differentiation 領域之一。
Frame:ATTRIBUTE· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
One of HBM’s key challenges is delivering power up the stack with power TSVs. Refresh operations especially draw a lot of power, and the design of the power distribution network is important. Hynix’s HBM3E has reduced peripheral area and introduced all-around power TSVs on the die instead of having two banks of power TSVs, thereby increasing the number of TSVs by almost 6x. As a result, SKH has achieved a much lower IR drop, up to 75% lower for VPP.
Atomic Claim 21/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0021
Claim: Hynix HBM3E 採 all-around power TSV，使 TSV 數量接近增加 6 倍，VPP IR drop 最多降低 75%。
Frame:ATTRIBUTE· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Similarly, Micron’s surprising leapfrog in HBM technology (Micron didn’t even offer standard HBM3) was due to its focus on TSVs and the power delivery network. The TSV network seems likely to be the point of differentiation that allows Micron to claim 30% lower power consumption, though that claim is yet to be verified.
Atomic Claim 22/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0022
Claim: Micron 將約 30% 低功耗歸因於 TSV/PDN 差異，但 SemiAnalysis 指出該 claim 尚未驗證。
Frame:ATTRIBUTE· Mode:ATTRIBUTED· Mapping:COMPLETE
開啟逐條審核
Source: Micron
The other thing is delivering the promised speeds within the power and thermal envelope. As with any 3DIC assembly, thermal dissipation is an issue, and DRAM especially doesn’t like heat. As data from hyperscalers have shown, HBM failures are the number one cause of GPU failures, which happen more frequently than other chips in the data center.
All manufacturers have absolute yields well below what they’re accustomed to compared to their conventional memory wafers, so it is a question of relative yields and the end economics. For SK and Micron, yield loss is more than made up for by high pricing, and hence, HBM is margin accretive. For Samsung, yields are even worse. Ironically, their low yields tighten up the total DRAM wafer supply, leading to higher pricing.
This leads us to layer counts. Higher layer counts are harder to achieve. Simplistically if the stack yield of a single layer is x%, each layer’s yield will accumulate to x% to the power of n bond steps (which is just total layers less 1). Take an 8-layer stack with 99% stack yield per layer; total yield will be 92%. For a 12-layer stack, this becomes 87%. Of course, this is overly simplified. Yield degrades at higher layer counts as non-critical stack defects can accumulate. For example, a small but acceptable amount of non-co-planarity at a few layers can result in an unacceptable amount of co-planarity at higher levels.
Process Flow: Bonding Tools, SK Hynix + Hanmi Drama
The bonding or die attach step is a key contributor to yield and thus requires sophisticated tools. With TSV pitches around 40 µm, the bonders must be capable of single-digit or even sub-micron alignment accuracy. Even pressure distribution is also crucial to avoid warpage that will compound over many layers. And throughput of course is important as it drives cost.
Hanmi made an early bet to focus on thermocompression (TC) bonders for HBM, a segment at that time ignored by market leaders Besi and ASMPT. This paid off in a near monopoly in current HBM processes. At SK Hynix their share was 100% until last fall, when Hynix placed a large order with competing tools from Hanwha. Allegedly, they paid Hanwha a higher price for the tools.
This set off a firestorm at Hanmi who, understandably, were upset to see a competitor winning with higher prices despite not being qualified on Hynix’s process for HBM supply to Nvidia, the largest and most important HBM customer.
A proud Hanmi employee stands next to a TC bonder in his cleanroom suit. (source: Hanmi)
The dispute reached a fever pitch in early April when Hanmi pulled its field service teams out of SK Hynix fabs. Without service, it would be months if not weeks before Hynix was unable to ship its marquee products. Longer-term it would threaten the entire accelerator supply chain as Micron and Samsung would not quickly be able to fill the capacity void. Hanwha’s tools had not yet been delivered and a batch of ASMPT bonders ordered last fall do not work for Hynix’s HBM3E 12-high. This left the fabs with little choice but to beg forgiveness from Hanmi.
Under enormous pressure, SK Hynix placed a small order with Hanmi in recent weeks. It appears this was more to placate Hanmi than a large volume order, but it was enough to restore field service to the tools. There may not be much more juice for Hanmi to squeeze from their monopoly power, as ASMPT, Besi, and others are rushing to improve their HBM-specific TC bonders.
China: CXMT & Huawei HBM
Export restrictions ban the transfer of all raw HBM stacks into China; however, chips with HBM can still be shipped as long as they don’t exceed the FLOPS regulations. Currently, banned HBM is still being reexported to China through a network involving CoAsia Electronics, Faraday and SPIL ↗ which allows end users in China to desolder and reclaim the HBM from GPU packages.
As HBM is one of the key ingredients for accelerators, and with export restrictions threatening to cut off HBM supply, China is naturally pouring resources into domestic development efforts. China has planned 200B USD in subsidies for home-grown semiconductors over the next five years. Expect a material portion will go towards HBM. DRAM national champion CXMT is aggressively expanding HBM capacity, with a large stockpile of tools to insulate from updated export controls (U.S. increased HBM controls in Dec. 2024, Korea more recently). HBM2 8-high will enter mass production in the first half of 2025, with TSV capacity matching Micron’s by the end of the year.
Huawei, which has never seen a high-tech market it doesn’t want to enter, has its own HBM affiliates with XMC (Wuhan Xinxin) producing the HBM wafers and SJSemi (Shenghe Jingwei Semiconductor) packaging them. Current capacity is at R&D scale, not high volume, but is planned to ramp in coming years. Both XMC and SJSemi are entity listed and thus restricted from buying equipment with U.S.-origin content (GlobalFoundries recently got a slap on the wrist despite selling more than $17 million in chips to SJSemi without a license).
XMC’s DRAM / HBM fab, supplying Huawei’s domestic HBM efforts. (Source: Google, SemiAnalysis)
HBM Stack Counts – Let’s Get High, to Hybrid Bond or Not?
More layers in the HBM stack means more memory capacity. With each generation, layer counts have gone higher. This stack height has been contained within a 720-micron height cube (the current JEDEC standard) up until now. To fit in more layers, each die (excluding the top die, which needs to be much thicker to withstand handling during subsequent packaging steps) has been made thinner, and the bump gap between the dies has also been made thinner to create more space to fit in more layers. Thinner dies are increasingly difficult to handle and, therefore, more vulnerable to warpage and breakage, which hurts yields.
The main benefit of Hybrid bonding (HB) for HBM is it is bump-less. By eliminating the bump gap this frees up room for more DRAM core layers to fit. This introduces a whole host of new challenges for yield and cost ↗ which may not be worth it, especially as HBM doesn’t need the level of interconnect density that hybrid bonding offers. If anything, manufacturers are exploring how much the pad density can be relaxed in hybrid bonding to ease bond accuracy requirements to make the technology more viable for HBM. ↗ HB also offers a one off power and heat benefit but the primary benefit is stack height.
HB adoption for HBM has always been a “next generation” technology and the goalposts continue to keep shifting. For D2W hybrid bonding, getting yields to be acceptable is extremely difficult and expensive for a 2 layer. Imagine scaling that problem for 16 layers and higher. The memory players are still very early in their hybrid bonding technology development. In TSMC’s experience, HB adoption has taken a long time to reach volume production and even when the performance benefits are clearer in advanced logic - adoption has taken a long time.
HBM3 and HBM3E are going up to 12-hi stacks and with bump-based interconnect, 12-hi has approached the limit within the current 720um cube thickness. The two solutions to go higher are either bump-less or making the stack taller/thicker. In a blow to hybrid bonding adoption, the latter has been chosen with JEDEC confirming a relaxation to 775 um stack height.
Height relaxation can go even further. 775um is the standard thickness of a silicon wafer. HBM needs to be the same height as the logic die it is co-packaged with. To have something higher than a 775 um stack, the logic wafer would also need to be thicker and the current equipment is not designed to accept thicker wafers. One potential solution could be to raise the molding of the interposer underneath the logic to raise it and ensure coplanarity with the neighboring HBM, though this lengthens the traces and implementing silicon bridges for interconnect will not be straightforward.
Source: SemiAnalysis
While initially there was more talk about HB implementation for HBM4, this has been shifted back to 4E. Recently, both Hynix and Micron have been far quieter on HB adoption, while Samsung is the loudest. This is typical for Samsung which often promotes the most aggressive technology implementations in attempts to catch up, only to expectedly fail on execution. This leads to them falling even further behind.
While 16 layer is solved by the higher stack height, to get to 20 layer and beyond, there is likely a further reduction of the bump gap and more wafer thinning, or we simply cap out at 16 layers. With higher stacks you can get more density but more stacks offer bandwidth and density.
Throughput Optimized: I/O is the Lifeblood of the AI Accelerator
The key defining feature of an AI accelerator is they are highly parallelized and optimized for throughput. Accelerators are designed to maximize the total number of operations an accelerator can perform per second by sacrificing the complexity of these operations. Most accelerators focus on multiplication and addition operations for General Matrix Multiplication (GEMMs), which dominate AI training and inference workloads. This compares to CPUs, which are also focused on how many instructions per second they can execute, but CPU cores are much “smarter,” which requires far more circuitry and area. Therefore they are designed to execute a broad range of more complicated tasks but at much lower throughput.
The implication is that AI accelerators need a lot of off-chip bandwidth for memory and the scale up and scale out fabrics. Bandwidth is needed to move the processed data off-chip whilst also feeding the accelerator unit with more data to process. Without sufficient bandwidth, the XPU’s compute elements will be left unutilized, defeating the purpose of having all this parallel computing power available. Let’s start with memory needs
Memory content increases
To deliver more performance - increasing memory capacity and bandwidth along with FLOPs has been paramount and easily observed in accelerator roadmaps. There are 3 dimensions where capacity and bandwidth are scaling:
Newer generations of HBM deliver higher bandwidth through faster signalling speeds, and denser core die
Adding the number of layers per stack increases capacity. We are on the cusp of 12-high HBM becoming the mainstream configuration
Adding more HBM stacks per package adds more bandwidth and capacity
Here we can see this in Nvidia’s roadmap. HBM capacity explodes from the A100’s 80 GB of HBM2E to a 1024 GB of HBM4E for Rubin Ultra. Memory bandwidth per chip also increases dramatically. From Ampere to Blackwell Ultra, the biggest absolute and relative increase in the bill of materials comes from the additional HBM content- benefitting the memory vendors (primarily SK Hynix).
Atomic Claim 23/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0023
Claim: Nvidia roadmap 中，HBM capacity 從 A100 的 80GB HBM2E 增至 Rubin Ultra 的 1,024GB HBM4E。
Frame:COMPARISON· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Source: SemiAnalysis
This is related to the need for non-memory I/O as well. Scaling up more GPUs in a single memory-coherent domain delivers more aggregate memory capacity and bandwidth. This allows scaling inference of larger parameter models and support for much longer context lengths that are becoming prevalent in reasoning models and complex workloads.
Just as Parkinson’s Law observes that work expands to fill the time allotted, modern AI follows a “memory‑Parkinson” dynamic in which neural‑network architectures relentlessly grow to occupy whatever HBM becomes available. Each generational bump in HBM capacity and throughput, whether 80 GB at 3 TB/s on H100 or 192 GB at 8 TB/s on GB200, quickly encourages designers to increase parameter counts, context lengths, and KVCache footprints, nullifying the headroom that seemed ample only months earlier. Techniques once deployed to squeeze models into tight budgets (activation checkpointing, optimizer off‑loading, weight quantization) are relaxed as soon as new HBM space appears, until the memory wall is hit again and efficiency tricks must be rediscovered. In effect, the mere presence of larger, faster HBM does not yield sustained slack; instead it resets the baseline for “reasonable” model size, ensuring that capacity and bandwidth remain limiting factors despite silicon advances. Essentially, as AI chips get more HBM, developers immediately build larger models to fill it, so memory is always the next bottleneck. Let’s go through how HBM is used, and where the pressures are.
HBM Usage in Inference
In LLM inference, all the model weights reside permanently in the on-package HBM memory so the GPU can fetch them without delay. Alongside the weights, HBM also holds the KVcache. Each time the model is asked to produce the next token, the GPU first reads the weights from HBM and simultaneously retrieves the entire KV cache so it can compare the new token against the conversation history during the self‑attention phase. After computing, the GPU appends a fresh key and value for the newly generated token back into HBM, enlarging the cache. This is very demanding on bandwidth as every token decode step repeatedly reads both the static weights and the ever‑growing KV cache. If memory bandwidth is not able to move this data at terabytes per second the GPU spends more time waiting for memory than performing computation. This is what happens in reality, as bandwidth significantly overwhelms the compute intensity of token decode, making most LLM inference workloads memory bandwidth-bound rather than compute bound.
As models improve, they have increased in horizon lengths. What this means is that models are able to think, plan, and act for longer periods of time. This rate of increase has been exponential and has already manifested itself in superior products. Deep Research from OpenAI, for example, can think for tens of minutes at a time, while GPT-4 mustered mere tens of seconds.
As models can now think and reason over a long period of time, the pressure on memory capacity explodes as context lengths regularly exceed hundreds of thousands of tokens. Despite recent advances that have reduced the amount of KVCache generated per token, memory constraints still grow quickly. One way to deal with this has been to serve reasoning models at lower batch sizes which is harmful to economics.
The main driver of AI progress is Reinforcement Learning (RL) and a huge part of the RL paradigm is inference. As an example, often what is needed for RL is synthetic data satisfying strict requirements, which means conducting many GPU-hours worth of inference to generate data that is later filtered by another model. Another example of heavy inference loads is RL for hard-to-verify tasks like creative writing. Unlike code, which can be checked and verified easily, things like creative writing, legal work, and teaching cannot be verified trivially. The way to get around this, and thus to get a signal to reinforce and improve the model, is to have another model judging the answers. This LLM-as-a-judge is then given a rubric, currently hand written but soon to be automated by LLMs, which it uses to grade the answer.
KVCache offload
There are various algorithmic or setup improvements that are designed to reduce pressure on scarce HBM. One technique is offloading the KVCache to a cheaper and more available tier of memory such as conventional DDR or even storage.
Today, KVCache offloading is already commonly used . Nvidia has a framework for this called Dynamo Distributed KVCache Manager. Conceptually this is not so different from the multiple tiers of memory available in general purpose CPUs: extremely fast but low density L1/2/3 Cache and slower but high density DRAM. In an AI system, where the where the KVs are stored based are managed based on frequency of usage. A well-optimized system keeps all currently used KVs in HBM, infrequently used KV in DDR, and very rarely used KV in NVMe.
Just like how DRAM is not cannibalistic to L1/L2/L3 cache demand for CPU, HBM and DDR/SSD offload do not compete with each other directly. In fact, for most modern LLM workloads, the prefill speed (aka the rate at which KVCache is produced) is typically slower than the transfer rate to DDR or NVMe SSD, meaning KV rarely ‘lives in HBM’ in their entirety. They get produced and evicted or sent to the decode node to be used to produce the next token. Mostly it’s the system prompt that is used for every user that is kept in HBM as well as other hot KVs such as the active sequence window and some prefetch buffers.
As for whether DDR or NVMe is used, it depends on workload needs and size of workload. It also depends on how often the workload is cycled as frequently cycled KVs are not a good fit for the limited write/rewrite tolerance of NAND. Agentic use cases that use tool calling to pull in documents and data at very low latency and high cycle rates today further push caching from NVMe to DDR. These are architectural and user experience tradeoffs to consider, rather than direct substitutes to each other.
As use cases evolve, there may be different hardware setups used for different inference needs. For example, querying a fixed code base or document would benefit from accessing larger amounts of KVCache per user per GPU as the memory demands of those user behaviors is extremely high compared to normal chat.
HBM for Pre-Training
For traditional pre-training, everything the GPU needs for a forward‑and‑backward step passes across HBM. First, the model’s weights are stored in HBM so each layer can read them quickly while computing the forward pass on a batch of data. As every layer processes the batch, it writes intermediate activations into HBM so they are available later. Once the forward pass finishes and the loss is calculated, the backward pass begins: the GPU revisits those stored activations and weights, reading them from HBM to compute gradients. The resulting weight gradients, plus any auxiliary optimizer statistics (for example, momentum or variance terms in Adam), are also written to HBM. Finally, the optimizer reads those gradients and statistics from HBM to update the weights in place, ready for the next iteration. However, training operations require more compute relative to data transfer, meaning training is more often compute bound. But as mentioned above, RL is now key to improving model capability - so what was traditionally achieved with pre-training is becoming more like inference in the form of reinforcement learning.
The rest of this piece will discuss OpenAI’s ASIC project, shoreline area challenges, the revolutionary change coming to HBM4 with custom base dies for HBM, the advantages on PHYs, memory controller offload, repeater PHYs, LPDDR + HBM combos, and various beachfront expansion techniques. It will also discuss SRAM, compute under memory, supply chain implications, and Samsung’s struggles.
Bandwidth beats capacity
Although every accelerator has strived to design in the most HBM that is attainable, we understand that OpenAI’s ASIC project is going to break this trend. OpenAI is choosing to use 8-Hi HBM4 instead of opting for 16-Hi or even 12-Hi HBM. 8-Hi has been expected to be phased out as it is not mentioned on the major memory vendors’ roadmaps. 12-Hi becoming the standard from as early as next year. This is notable as the first time a customer that is aggressive on silicon has requested a downgrade in specifications.
Atomic Claim 24/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0024
Claim: SemiAnalysis 表示 OpenAI ASIC 將選擇 8-Hi HBM4，而非 12-Hi/16-Hi HBM。
Frame:NARY_RELATION· Mode:EXPECTED· Mapping:COMPLETE
開啟逐條審核
This is because OAI sees 8-Hi offering a much better ratio of bandwidth to capacity at the given cost. Capacity is important, but for inference bandwidth is most often the constraint. With 8-Hi stacks, OAI gets the same bandwidth but at less than half the price per stack. The demands on memory capacity is relieved by the direction of OAI’s inference setup involving high-rank expert parallelism. This involves sharding the individual model experts across a much larger number of GPUs. Spreading the experts thinly leaves much more memory capacity available for KVcache while bandwidth is still the constraint. This is not a free lunch and there are tradeoffs in this equation: increasing world size and sharding workloads across many GPUs is more taxing on the network, a trade-off now faced across the entire industry as more firms adopt high-rank expert parallelism, and this underscores why I/O remains fundamental to everything.
Atomic Claim 25/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0025
Claim: OpenAI 認為 8-Hi HBM4 在 inference 中可維持相同 bandwidth、但每 stack 成本不到一半，因此 bandwidth/capacity/cost ratio 更佳。
Frame:ATTRIBUTE· Mode:ATTRIBUTED· Mapping:PARTIAL
開啟逐條審核
However, this shouldn’t be interpreted as a leading lab calling time on the trend of HBM capacity scaling. OAI will focus more on improving the software, micro-architecture, networking within the broader system so that future generations can become performace/TCO competitive with merchant solutions. Reducing memory costs is a good way to reduce the investment without compromising on the other surface areas for improvement and adding more capacity with higher layers is relatively trivial. OpenAI will still be dependent on GPUs in that timeframe. Rubin Ultra will be there with abundant capacity as model architects will no doubt try to find ways to consume this additional capacity to extract more intelligence. Going for lower capacity is a cheaper, and relative lower risk way for OpenAI to explore different cost and performance tradeoffs for their accelerator.
Ultimately HBM demand will be influenced by both user behavior and the architectural decisions of accelerator designers, ie what they consider is optimal in terms of HBM memory capacity / bandwidth vs FLOPs over the lifetime of the chip. With a lifecycle of 4 years, designers need to take into account the evolution of workloads over that period and make design choices that are flexible enough to adapt to different needs of model architectures for inference and training.
Adding more valuable shoreline real estate
We’ve spoken about the important of bandwidth - so why is it hard to just add more?
The amount of total I/O is constrained by the chip shoreline or chip “edge”. This edge is valuable and where the off-silicon I/O-related circuitry is located. This typically include memory controllers (transferring data between host and DRAM) and various forms of SerDes such as the ubiquitous PCIe protocol: a standardized and versatile protocol used to transfer data to other XPUs, storage, NICs in a system.
The performance of various interface protocols can be judged by relative shoreline density for a given reach. More performant PHYs can offer more bandwidth for each valuable mm of valuable shoreline the PHY circuitry takes up. It is not a coincidence that the leaders in AI chip shipments also have the best high speed networking IP: this is a necessity for throughput driven systems. This why hyperscalers tend to work with Broadcom and Marvell to get access to their high speed SerDes IP.
To get more I/O, maximizing both total shoreline area and density is key. The availability of enough shoreline itself is a key design consideration influencing package design. Given the endless desire for more memory bandwidth and capacity, accelerator designers opt for trade-offs that balance the shoreline available for HBM and off-chip IO.
Source: SemiAnalysis
A 4 SOC configuration with a 2x2 reticle size die in a square grid like we see in the diagram is not ideal as the available shoreline relative to silicon area is lower than if the compute die was laid out in a single row. In this case, there is only enough shoreline for 8 HBM, like a dual SOC MCM like Blackwell – halving the available memory bandwidth and capacity relative to compute.
Source: SemiAnalysis
This is a sub-optimal trade-off given AI workloads are often memory bound rather than compute bound. This is why the Rubin Ultra package is more oblong shaped with a single row of SOCs. This increases edge and allows for double the HBM sites.
Source: SemiAnalysis
The importance of I/O and shoreline has a few major implications on HBM configurations given the need to maximise this limited shoreline:
Use the fastest and densest HBM feasible and/or available in the supply chain
HBM content per accelerator is constrained by available XPU shoreline
Within HBM itself there is undeveloped shoreline that can be unlocked from HBM4 onwards
The Revolutionary Changes Coming with HBM4
HBM4 is where there will be more drastic changes to the HBM architecture on many facets: bus width, form factor and customization vs. “standard” off-the-shelf HBM.
Atomic Claim 26/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0026
Claim: HBM4 將在 bus width、form factor 與 customization 等面向帶來較大架構變化。
Frame:ATTRIBUTE· Mode:EXPECTED· Mapping:COMPLETE
開啟逐條審核
HBM is high bandwidth because it has a much wider memory bus of 1024 bits compared to other forms of DRAM with 64 bits (16x the I/O). The bus of a single stack consists of multiple channels on each of the HBM core die. Currently, the bus for HBM3 and HBM3E is divided into sixteen 64-bit channels, with two nearly independent pseudo-channels per channel.
The big change for HBM4 is that the bus width will double to 2048-bit wide from the 1024-bit bus. Memory manufacturers will be relying more on the wider bus to drive up bandwidth rather than increasing pin speeds. JEDEC has said there is an initial agreement on speeds up to 6.4Gbps implying up to 1.6TB/s which is around 1.5x the bandwidth of HBM3E at peak speeds of 9.2Gbps. The wider bus will require more TSV area so the die size will increase 20% over the HBM 2E-3E size.
Atomic Claim 27/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0027
Claim: HBM4 bus width 將由 1,024-bit 加倍至 2,048-bit，memory vendors 更依賴 wider bus 而非提高 pin speed 來增加 bandwidth。
Frame:ATTRIBUTE· Mode:EXPECTED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 28/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0028
Claim: JEDEC 初步 HBM4 規格為最高 6.4Gbps、約 1.6TB/s，約為 HBM3E peak bandwidth 的 1.5 倍。
Frame:COMPARISON· Mode:ATTRIBUTED· Mapping:COMPLETE
開啟逐條審核
Source: SK Hynix
Custom Base Die
Today, the base die in HBM is simple: manufactured internally with a DRAM process with regular planar CMOS transistors that is equivalent to at best a 22nm node. Even though HBM is a premium, high-performance product, the base die implementation is anything but: only aiming to achieve the bare minimum to interface between the memory and the host XPU. These lower performance transistors hamper the signaling capability of PHY so that signals can only be fixed and unidirectional at slow speeds. The standard PHY is much slower than the state of the art PHYs we see in leading edge logic.
The base die is slightly larger than the memory die at around 105mm2, but only half of the area is actually utilized to provide sufficient (but far from optimal) functionality consisting of: the PHY that interfaces with the host XPU, TSV banks in the middle, and then area reserved for Direct Access Balls that are used for testing.
This is the basic approach. While HBM is a high-performance premium product, the base die implementation provides the minimum required functionality and nothing more. This is a consequence of the cost driven mindset that is still the prevailing consideration for memory manufacturers. There is a lot of under utilized potential in the HBM base die that offers a lot of potential and with HBM4 and beyond, that potential will become realized.
Hynix’s HBM3E base die. Source: SK Hynix
First, the base die will move to a modern logic process. Samsung can manufacture this with their logic/foundry capability, but this is beyond what the pure memory manufacturers like Hynix and Micron can do internally, so this will be manufactured by TSMC. TSMC has confirmed there will be 2 variations of the HBM4 base die, one manufactured on N12 and one on N3, this would offer a significant power reduction compared to existing implementations (though a very small reduction in overall power budget of the system). Hynix and Micron also do not have leading edge logic design capability, so they will need the help of advanced logic design houses or IP vendors to contribute to these designs.
Atomic Claim 29/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0029
Claim: HBM4 base die 將轉向 modern logic process；Hynix/Micron 需依賴外部 foundry，SemiAnalysis 指向 TSMC。
Frame:NARY_RELATION· Mode:EXPECTED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 30/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0030
Claim: TSMC 確認 HBM4 base die 將有 N12 與 N3 兩種版本，base die power 可顯著下降，但對整體 system power budget 的改善較小。
Frame:ATTRIBUTE· Mode:ATTRIBUTED· Mapping:COMPLETE
開啟逐條審核
More of the area of the base die can be utilized to add in significant functionality with various custom base die configurations. Tying this back to the discussion on the importance of shoreline -** there is a huge amount of untapped shoreline real estate that exists on the HBM base die of today which is going to be unleashed from HBM4 onwards.**
We expect Nvidia and AMD to come out with their custom HBM implementations with HBM4, with other accelerator designers likely to only have a custom base die implementation by HBM4E in the 2027 timeframe.
Atomic Claim 31/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0031
Claim: SemiAnalysis 預期 Nvidia 與 AMD 會從 HBM4 導入 custom HBM，其他 accelerator designers 可能到 2027 HBM4E 才導入 custom base die。
Frame:NARY_RELATION· Mode:EXPECTED· Mapping:COMPLETE
開啟逐條審核
We believe these custom base die implementations will provide 3 main functions: memory controller offload, shoreline expansion, and compute in base die. Let’s examine the possibilities.
Atomic Claim 32/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0032
Claim: SemiAnalysis 預期 HBM custom base die 的三大功能為 memory-controller offload、shoreline expansion、compute in base die。
Frame:ATTRIBUTE· Mode:EXPECTED· Mapping:COMPLETE
開啟逐條審核
Getting Down to the Wire: Better PHYs
At the heart of the problems with the current base die design is the PHY that interfaces with the host. HBM uses a standard DDR PHY using planar transistors fabricated on a DRAM process. These lower performance transistors hamper the signaling capability of PHY so that signals can only be fixed and unidirectional at slow speeds.
UCIe, for example, offers bandwidth density of 10 Tbps/mm, which is almost 15x the beachfront efficiency of HBM3E (~0.7 Tbps/mm) while also using less energy per bit. UCIe is often derided as too slow of an interface - which tells you how far behind the state-of-the-art DDR PHY is.
Atomic Claim 33/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0033
Claim: UCIe bandwidth density 約 10Tbps/mm，約為 HBM3E ~0.7Tbps/mm beachfront efficiency 的 15 倍，且 energy/bit 更低。
Frame:COMPARISON· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Companies like Eliyan are publicly pitching their NuLink interconnect for a role in HBM4 base dies, so there may be multiple options. We can reasonably see UCIe as the baseline improvement likely to be delivered, since it has already proven that 10x better beachfronts can be in production.
Marvell is the first company to reveal their custom HBM base die solution with the main improvement via implementing a D2D PHY that uses only 1/4 of the shoreline, freeing up silicon for other uses.
Hynix’s HBM3E base die. Source: SK Hynix
Currently HBM3E expends 95% of energy per bit on moving the data and the I/O interface, and less than 5% into reading or writing values in the DRAM. There is valuable low-hanging fruit from adopting a more efficient interface. We can see reduced energy per bit in a point-to-point connection rather than the half-duplex bus which has been traditional for DDR. Commands can be moved onto the fabric to reduce lane count. Read lanes may be more numerous than write lanes, reflecting the general bias towards more traffic in reads than writes.
Atomic Claim 34/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0034
Claim: HBM3E 每 bit energy 約 95% 用於 data movement/I/O，不到 5% 用於 DRAM read/write。
Frame:ATTRIBUTE· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
These improvements have been discussed for a long time in DRAM forums such as JEDEC but generally blocked by the slow performance of the transistors on DRAM chips. The vendors quite reasonably want to stick with the DDR style interface where they have 30 years’ experience of implementing it despite transistor limitations. However, when the base chip is in play those limitations no longer apply. The base chip might be in a reduced cost version of an advanced logic process like 5nm where clearly the advanced PHY options are all available.
Moreover, the base chip is situated right under the TSVs which total a fraction of a mm in height, a very short distance for a PHY to bridge. It should be possible for signaling systems at higher frequencies even with DRAM-process transistors, like the proven GDDR use of bit rates up to 32 Gbps, to drive faster data at much lower energy per bit over these short distances. There are some issues here in signal quality for TSVs in a tall stack, but clearly an incentive to find improvements that leverage a smart, modern base chip. HBM4 will be a phase change in how we look at connecting to DRAM.
Memory Controller Offload
The GH100 GPU die has 12 HBM controllers on the side that interface and control the HBM adjacent. Source: Nvidia
The other easy win would be to offload memory control functionality onto the base die. Memory controllers are typically found on the host XPU and are responsible for interfacing with external memory. The controllers issue read and write instructions from the host. There are also multiple commands to control the memory cells for every single transfer – equalization, activation, sense enable, data transfers (sequentially repeated at 1ns intervals), and shut off. These control commands are independent of the host. These operations consume power and set up competition between operations due to the shared interface lines as the DRAM control instructions are sent from the host chip to the memory chip.
Architectural block diagram of a HBM3E controller. Source: Rambus
In theory, the host is a poor place to put the memory controller: the host uses the most expensive silicon, and it is distant from the DRAM chip. In practice this is where the memory controller has been implemented due to the complications of fabricating advanced logic circuitry alongside DRAM cells. ↗ Of course the next best thing is HBM has a logic base die underneath, and from HBM4 onwards that base die will be fabricated with an advanced logic process. This facilitates an alternative and better approach which is to extend the host fabric across the external interface. The simple load and store commands can be delivered from host to the HBM base chip, and then the base chip with its cheaper logic and closer integration to the DRAM is in a better position to perform DRAM control with less bottleneck and less energy used.
The memory controller on the base chip obeying the load and storing semantics has a lot of flexibility in implementation. The fabric allows flexible timing permitting caches which can accelerate some of the reads and hide the latency of some of the writes. It can even allow the base chip to work on a larger address space split between a first tier of the HBM chips directly above, with a second tier beyond the HBM package.
Repeater PHY – Beachfront Extension: Second Rank HBM or LPDDR
In a classic HBM configuration the HBM stacks are immediately adjacent to 2 sides of the XPU and crowd out all other use of those 2 edges of the XPU. The other 2 edges are required for other I/O like PCIE or custom protocols like NVLink or ICI. As mentioned above, this makes chip shoreline area very precious and is one of the constraints limiting total off-chip bandwidth.
Maximizing beachfront is a top consideration when it comes to layout and package design. When we look at shoreline in the traditional way as the perimeter of the processor, then designers have used up that to the full extent. However, beyond the existing shoreline, there is a pristine, undeveloped shoreline within the HBM base die. When we zoom out and think of the HBM base dies not just as merely the PHY for the DRAM that sits above it but more as logic chiplets that can have functionality within the MCM, the idea of a second shoreline on the outside perimeter of the HBM becomes apparent. This opens up a world of possibilities for interesting accelerator designs and capabilities.
LPDDR behind
One highly desirable use of the HBM base die is the ability to expand beachfront by adding a second PHY to interface with a rank of memory on the side facing away from the XPU.
Source: SemiAnalysis
The first implementation of custom HBM base dies will be seen in Nvidia’s Rubin and AMD’s MI400 platforms to add off-package LPDDR as second tier memory. This is effectively using the free area in the HBM base die and putting LPDDR memory controllers on the outward-facing edge, which doesn’t sacrifice valuable XPU shoreline. The PHY that interacts with the CPU host will also need to be able to pass through data and instructions intended for the LPDDR to this LPDDR controller. The benefit of LPDDR is cost: $/GB is 5x less than HBM, and it doesn’t need to be on the package (either substrate or interposer). Still, it can simply be on the same compute board of the XPU with data transfer occurring out of the substrate and via PCB traces to a swappable LPCAMM module.
Source: SemiAnalysis
In principle all this implementation requires is that the base chip includes logic, to recognize addresses that are not local to the stack above, and to pass the associated operations through to the second rank controllers with base die space reserved for a PHY at the far side of the chip to interface with the LPDDR that is outward from the XPU side. This adds a significant amount of memory capacity but with much lower bandwidth. This low rank second tier memory can be used to offload from HBM non-bandwidth sensitive tasks such as activations, some KVcache and certain post-training tasks. And post-training, the area which the labs are working the hardest on, is mostly memory bound.
Adding extra LPDDR behind HBM creates a second tier of memory that increases overall capacity, allowing the system to store larger KV caches and intermediate states without solely relying on the limited high-bandwidth HBM. LPDDR does not need to be high bandwidth, as it is used as a temporary storage for KV cache when users are typing their response. Typically, the user’s KV cache will stay in the HBM or in the CPU memory. However, with LPDDR behind, KV cache will be stored in the LPDDR while the user is typing their response and shifted to HBM when decoding happens. This additional memory tier helps accommodate the increased memory requirements from longer context lengths, paired comparisons in DPO, and the overhead from policy and verification models in RLHF, all while balancing cost and power efficiency.
HBM Behind
The more advanced extension of this is to daisy-chain another rank of HBM chips behind the first rank. The HBM behind will share the high bandwidth of the initial stack. This would dilute the bandwidth per GB but at least the bandwidth being shared by the two ranks will be the maximum edge bandwidth the XPU can handle with a DDR-style PHY. This is adding HBM capacity but without additional bandwidth - the same as adding more layers in the stack.
Source: SemiAnalysis
In principle all this requires is that the base chip includes logic, to recognize addresses that are not local to the stack above, and to pass the associated operations through to the second rank with some base die space reserved for a PHY at the far side of the chip to interface with the second rank memory that is outward from the XPU side. This will cause a mild NUMA effect as the second rank has slightly higher latency, but doubles the memory capacity.
If the throughput is exactly equal to a single HBM stack then this will likely just use a classic DDR style bank selection, like multiple DIMMs on a channel, rather than any significant intelligence on the base chip, mostly some form of redrive with direction (half duplex) awareness. This is a lot less base chip wizardry than would be needed, say, to chain to LPDDR6.
Source: SemiAnalysis
In principle, the second rank could daisy chain to another row as well, but there are diminishing returns on cost as the edge area bandwidth gets saturated and the bandwidth per GB gets diluted even further.
With only half the throughput going through to the second row of HBM, this could relax bit width requirements and perhaps with a slightly improved PHY here this HBM could be put on the substrate instead of on an interposer. This will reduce packaging cost and along with other considerations like the challenges of scaling interposer size, the relaxing of this could make this an attractive alternative to having a much taller stack of HBM to get equivalent density: Eg. 2 rows of 8-Hi vs a single row of 16-Hi.
With a much better PHY, aggregate bandwidth could be improved with the 2 rows of HBM individually having equivalent or higher bandwidth than standard HBM next to the shoreline. This is more challenging on base die design requirements, but this is how you get to doubling HBM content per chip with the same beach front area.
Beachfront Expansion: I/O Expansion
In an AI accelerator, with 2 sides of the beachfront often used up by the HBM, the remaining 2 sides are used for off-chip I/O: as we can see in the GH100 example above, one edge is for the PCIE interface, and the other is the NVLink interface. As with memory, off-chip bandwidth is also constrained by this beachfront constraint, and even more so in AI accelerators where half or more than half is dedicated to memory compared to other compute engines that have less memory dedicated shore front.
In a similar concept to having second tier memory behind an extension of this idea is that the shoreline can be expanded behind the HBM, to place IO chiplets behind them. The HBM base die will have bridges that pass data through to the IO chiplets behind. These IO chiplets could be various SerDes chiplets (such as for NVLink, ICI, UaLink, PCIE) or co-packaged optical engines for scale up over optics. This is one of the very promising ideas that we first saw demonstrated by startup Eliyan ↗. It is implemented with Eliyan’s proprietary NuLink PHY and their proposed industry standard interface called ‘UMI’ (Universal Memory Interface).
Source: SemiAnalysis, Eliyan
These I/O engines or chiplets can be on substrate or if higher B/W is needed on the interposer. There are various configurations here, but the broad principle is it expands beachfront to add more I/O and offer greater flexibility on the mix of memory and off-package I/O. In the diagram above, we have illustrated one 224G SerDes chiplet behind every stack of HBM. Note that because we can put the SerDes chiplets behind, there is strictly no need to reserve 2 edges of the XPU for off-chip I/O so we can put HBM around the whole primary shoreline. Assuming each of these I/O chiplets has 8 lanes of 224G SerDes, with 12 of these chiplets that’s an additional 2.4TB/s of uni-directional bandwidth whilst also adding 4 additional sites of HBM, compared to what is becoming a conventional 2 SOC, 8 HBM configuration.
Source: SemiAnalysis
Indeed, this will require the use of a much more performant PHY to deliver data that is intended to go off-package through the SerDes chiplets and optical engine in addition to the data that is being read and written into on-package memory.
SRAM
The base die will be an older, cheaper process than the host XPU. However, SRAM density has changed little in recent nodes. For example, a 3nm base die would have very good SRAM density at lower cost than a 2nm XPU. What SRAM intensive functionality might be good on the base die if we have extra space to play with?
One possibility is to put cache tags on the base die when the HBM is used as a Tier 1 cache. We will describe more about this in the Repeater PHY section.
It is also possible that the SRAM is organized as a memory-side cache (MSC). An MSC is simpler than other caches because it does not participate in a coherency protocol. To the rest of the system, it simply acts like part of the HBM memory. The rest of the system believes that the data is at an address in the HBM – and the system’s coherency resources track that.
However, the base chip may in fact cache some data due to speedups like prefetch or delayed write. Prefetch can take advantage of a higher in-stack bandwidth to anticipate future requests from the XPU without committing them to the external interface. Delayed write can allow higher priority reads to move in front of a write, ensuring that the external interface meets its highest priority use even when there is lower priority writing queued ahead of it. Simple look-asides can ensure that any reads of addresses in queue to be written are satisfied correctly with the latest value.
SRAM on the base chip is an essential part of other functions such as offloading the memory controller, and running computations near the memory.
The base die is on the order of 120 mm2, to fit within the HBM standard package. As much as 40 mm2 might be useful for SRAM, allowing space for the Serdes at edge, the Serdes for the TSVs, and whatever control logic for the memory is to be included. With a typical density of 20Mb per mm2, when using standard IP for complete arrays including single bit error correction, that would allow about 80 MB of SRAM to be available for use. More, or less, depending on how much space the other logic needs – this number is just to get some perspective on the possible resource.
Compute Under Memory
Compute functionality on the base die will be limited by thermals. The multiple layers of DRAM stacked on top reduce heat flow, plus any hot spot computation leaking heat into the DRAM chips will increase leakage and raise problems with refresh rate and disturbance effects.
Only low‑power, bandwidth‑sensitive tasks make sense. A prime candidate is tensor re‑ordering—stride, flatten, reshape, and similar operations that AI workloads constantly perform. Instead of dragging an entire tensor across the HBM bus, the base die can read it from all banks into its SRAM, rearrange the data locally, and ship only the needed slice back to the XPU. That both frees up expensive SRAM and logic on the host and reduces traffic on a bus that is often the bottleneck in inference.
Several other data‑centric kernels fit the same pattern. Filters—think of a hardware “grep” for pattern matches or a vector‑similarity search for nearest neighbours—can be implemented with tiny state machines that run at memory speed while consuming very little power. Transforms that map values through dictionaries or extract JSON fields (analogous to Unix awk) likewise benefit from performing the text‑shuffling right next to the data. Lightweight block compression is feasible, although its benefit depends on how well the data compresses and whether the added complexity is justified. Finally, simple reductions—for example, 32‑bit integer additions that accumulate results in place—can exploit the full HBM bandwidth without blowing the thermal budget; floating‑point sums may be harder, but the concept shows how reductions over a 16‑GB address space belong exactly where the bandwidth is highest.
Economics and Supply Chain Implications
Given the functionality custom base dies can add, we expect to see them proliferate by the end of the HBM4E era. Custom base dies will have several interesting dynamics on an already complex supply chain.
First is who designs these advanced base dies?
The merchant giants like Nvidia, AMD, and Intel will design this themselves as they have the capability and vision on what configurations unlock the most value (at least for Nvidia). Meanwhile, we still expect the memory manufacturers to offer a standard off-the-shelf base die on a more mature process. Samsung has its own advanced logic foundry. Hynix and Micron, more advanced logic chips are beyond their process capability and will likely need silicon partners to help and foundries. Marvell has been the first out of the gate to announce their own HBM base die design that has been have been suggested as respective design partners for custom functionality.
What about hyperscaler ASICs? Hyperscalers need to keep up with the best merchant GPUs so they will also need to make the most of this functionality though the degree of adoption will vary. For example, Google builds systems with more focus on reliability and their TPU design philosophy reflects that with silicon choices not being aggressive to sidestep the high failure rates that plague chip designs that redline performance so they will not be aggressive on things like compute in base die.
This one more feature that ASIC designers will need to rely on their custom silicon partners to design the suite of the features above that best complement the architecture of each ASIC program. We can add HBM base dies as another surface area where design partners can add value in their ASIC designs. The other option would be to assemble various off-the-shelf IP blocks to create the custom base die functionality they want, for example using Eliyan’s UMI as an all-in-one PHY for beachfront expansion.
The other interesting element is custom base dies strengthen lock-in for memory vendor. Each HBM vendor has their own proprietary TSV layouts (see above) so each custom base die will need to be configured to match a respective memory vendor’s TSV layout: a custom base die designed for Micron HBM4E won’t be compatible with SK Hynix HBM4E. To qualify a second and third memory vendor then requires additional tapeouts of the base die which raises the cost of multi-sourcing HBM. This has disproportionate impact to less-resourced chip companies.
Atomic Claim 35/35 · 2025-08-12_scaling-the-memory-wall-the-rise-and-roadmap-of-hbm::HBM25-0035
Claim: Custom base die 會強化 memory-vendor lock-in，因各 HBM vendor 的 proprietary TSV layout 不同；為 Micron HBM4E 設計的 base die 不相容於 SK Hynix HBM4E。
Frame:COMPARISON· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
On the flip side this makes life even more difficult for memory vendors who are behind on their roadmaps like Samsung. Customers would not want to undergo the extra effort and investment to tape out base die designs compatible with Samsung given the understandable concerns around whether Samsung is on track to deliver competitive HBM4 and beyond.
For once, these memory vendors can have something resembling that of customer switching costs.
This makes the supply chain even more complex. In simple terms, an Nvidia GPU today is manufactured by TSMC packaging together with the logic which comes from a TSMC wafer and the HBM which comes from memory IDM that is purchased by Nvidia and consigned to TSMC. What does it look like for HBM where the base die is manufactured by someone else and designed by someone else.
The HBM customer will choose their base die configuration of choice from these options: 1) off-the-shelf from a memory manufacturer, 2) off-the-shelf from an advanced logic house, or 3) fully custom design from a logic house.
We believe options 2) and 3) will emerge as the most popular options: advanced logic houses already have the capability and IP to implement advanced base die functionality. However, this adds additional supply chain and logistical complexity. While the exact model is not settled, the easiest path is the accelerator designer / end memory customer (whether it be Nvidia, Broadcom, or the hyperscaler that is directly procuring HBM) will design their own base die or get it from a silicon partner. This base die wafer will be fabricated at TSMC on behalf of the base die designer. The accelerator designer will purchase this wafer from the designer, then we see that it is likely that the base die designer then consigns it to the HBM manufacturer (ie. SK Hynix) who then packages this base die with the HBM memory die. Consequently, this package is finally shipped back to the accelerator designer who then consigns it to TSMC to package with logic using CoWoS.
How does the memory supplier capture more value directly? HBM base die is beneficial for HBM demand and memory content which the memory players capture indirectly.
As per above, it makes HBM more performant which only unlocks more value. More specifically, HBM custom base dies allow dramatically more memory content per accelerator, both for HBM (unlocking shoreline constraints) and conventional DRAM like LPDDR.
However, despite this, memory players are getting more disintermediated here. Despite shipping a higher value end product, the memory supplier isn’t participating in this, and if anything, the value uplift is really due to others in the supply chain participating more actively. While this represents a paradigm shift in the memory industry and a chance for memory players to step up, in typical fashion, they are not running fast enough towards this opportunity to establish themselves and strengthen their position in the supply chain.
This is another case of memory providers not being able to capture the full value of their product. HBM is the most profitable memory product in history, and arguably it is Nvidia that is the most profitable HBM vendor in the world as they stack their hefty gross margin on the HBM they ship in their accelerators. The HBM boom is bittersweet in the sense it is a reminder that even when it is time for the memory vendors to shine, they are never the brightest star.
However, the memory vendors have power in the form of their packaging and TSV capability. TSMC has to manufacture the wafer, but the TSV formation will need to be done by the memory manufacturer along with the stacking. TSVs are critical to the performance of the HBM, and this is core IP for memory manufacturers. A wafer that is TSV’ed externally without the exact process recipe of the memory manufacturer will likely have compatability issues. This is similar for the stacking and bonding part. While few would argue that TSMC is the leader in advanced packaging for HBM, TSMC does have the experience with multi-layer 3DIC stacks that the HBM manufacturers have or will have.
This is where the memory manufacturer have leverage and they can exert this to their benefit, especially with only Hynix and Micron being viable suppliers. Micron and Hynix can claw back some uplift in value. At a minimum, they should be able to earn a higher gross margin on HBM4 with custom base die just by charging the same ASP as conventional HBM4. The margin uplift comes from not needing to fabricate the base die wafer themselves and having it supplied externally. While customers can push back by arguing that the suppliers COGS are lower, at the end of the day, they need the memory vendor to TSV and package everything to make it all work, with the extra margin being the “corkage” charge for BYO base die.
Source: SemiAnalysis
We also talked about TSV layouts being proprietary and the base die design needs to accommodate for those TSVs. If the memory vendors are aggressive, they can charge licensing fees to expose the TSV layout so that they can design it too. This is additional revenue but also raises costs of using custom bases and can ensure greater vendor lock-in given that raises switching costs too. Of course, this only works if every player does the same and there is no “defection” in a prisoners’ dilemma scenario.
Source: SemiAnalysis
Another implication is the rising design complexity. The “standard” accelerator in a few years will need 3 different dies or tapeouts at a minimum: Compute Die, I/O chiplets, Custom HBM die. There will be multiple HBM vendors and separate HBM base die tapeouts for each memory vendor to map to their TSV layouts. This adds greater challenges for the hyperscaler ASIC programs. To the current world of monolithic SOC + standard HBM, the hyperscalers do not have the internal resources to design, tapeout and bring a chip through to manufacturing by themselves. The additional complexity will only further swamp them and require more help from silicon design partners.
Samsung’s Struggles And Are They Back?
Similar to the story of the other large IDM, Intel, Samsung is losing on several fronts in memory and logic due to mismanagement and cultural rot. ↗
Samsung’s struggles in HBM come from to a combination of their poor front-end DRAM processes and packaging. This has resulted in awful yields and a worse performing end product. That’s why Samsung’s product doesn’t meet the requirements of more demanding customers such as Nvidia. While Samsung maintained a healthy market share up until HBM2E, Samsung’s process shortfalls started to come to light in HBM3, allowing Hynix to dominate share with the Nvidia Hopper which was around the dawn of the AI revolution. Samsung eventually did qualify their HBM3 but for Nvidia’s H20. Samsung did have a very technically demanding customer in the form of AMD for the MI300X. The MI300X was the first mainstream product that had 12-hi HBM and it was Samsung that was the sole supplier AMD chose. This proved to be a big mistake, as the Samsung HBM used was a significant contributor to GPU failures (well above the average) and one of the things that put a dent in AMD’s relationship with their major hyperscaler supporter Microsoft.
Things have since gotten worse with HBM3E. Samsung’s HBM3E is based on their 1a process node which was designed for mobile. Competitors SK and Micron use superior 1b processes for their HBM3E. As a consequence of a poorly designed node - Samsung’s 3E has further fallen behind in delivering the speeds customers within the necessary thermal envelope. The deteriorating performance gap relative to SK and Micron has caused share further share losses. AMD has shifted a significant part of their HBM3E supply to Micron. In the major hyperscaler ASICs, where Samsung has had a significant amount of share, are both moving away to Hynix to the extent they can. Amazon’s Trainium2e socket is the one where Samsung retains the most HBM3E share, with Amazon being more willing to sacrifice performance for lower cost.
Samsung hasn’t given up, and memory market watchers are looking closely at 2 things which will have an impact on HBM supply and demand dynamics. Despite repeated failures to qualify HBM3E with Nvidia, Samsung is still trying to qualify their 12-hi HBM3E after a re-design in the hopes of supplying Blackwell Ultra. Recent market chatter is noticeably more optimistic on Samsung’s chances for their newest round of qualification with some suggesting that Samsung has already passed. However, our understanding is that Samsung has yet to pass the full qualification. Nvidia is also well supplied with HBM3E this year. Although Nvidia still has demand for 3E 12-Hi for Blackwell next year, this bit demand will quickly shift to HBM4 as Rubin ramps.
Samsung is also enticing customers by offering very low pricing for all the inventory that is still on their balance sheet, which they have produced in anticipation of qualification.
Battle for HBM4
Following its disappointing HBM3E showing, Samsung is taking a bigger swing on HBM4 by jumping their 1c DRAM node, a riskier move aimed at closing the performance and yield gap. By contrast, SK Hynix and Micron plan to remain on 1b for their HBM4.
Aggressive attempts to leapfrog competitors are often troublesome (see Intel and Samsung Foundry), especially if the systemic issues that caused them remain unsolved. Samsung’s 1c could suffer from if the node inherits some of the fundamental issues with previous nodes. However, the alternative narrative is that Samsung is changing their priorities to performance first with less concern over cost which is fueling the hope that HBM4 can be a fresh start for Samsung.
So far, we understand that Samsung’s recent test production wafer yields are still extremely low, suggesting that so far not much has changed. Meanwhile, Hynix’s HBM4 is progressing smoothly, with yields the same level as previous generations during the same phase of development. Even if Samsung gets back on track for HBM4, they would only be able to supply meaningful volumes to customers later into 2026 based the schedule to sample, qualify and manufacturing cycle teams. However, in this scenario, that would be the first time all 3 memory manufacturers are back in the supply picture, with Samsung having a significant amount of TSV capacity that is effectively dormant due to yield and not enough demand.