SA Article Coverage Review · 2024-07-17_gb200-hardware-architecture-and-component
Coverage Summary
- Source: 開啟原始 SA 文章
- Atomic Claims:
45 - Source blocks:
468 - Blocks with ≥1 Atomic Claim:
24 - Blocks without Atomic Claim:
444 - Unplaced Claims:
0
Coverage Review
請從頭到尾閱讀下方 SA 全文。Atomic Claim 會依 Evidence 在原文出現的位置 inline 插入。
若該英文段落已有繁中翻譯 cache，翻譯只會作為淡色閱讀輔助顯示；不會進入 source、Claim provenance 或 Graph。
沒有 Claim callout 的段落不一定有問題；若內容重要且應形成知識，請記到 Missing Claim Notes。
Missing Claim Notes
- 若看到重要但沒有 Atomic Claim 的段落，請在這裡記錄：
- Section:
- Evidence:
- 為什麼重要／應該抽成什麼 Claim：
SA Full Text + Translation + Atomic Claims
GB200 Hardware Architecture - Component Supply Chain & BOM
Nvidia’s GB200 brings significant advances in performance ↗ via superior hardware architecture ↗, but the deployment complexities rise dramatically. While on the face of it, Nvidia has released a standard rack that people will just install in their datacenters without much trouble, plug-and-play style, the reality is there are dozens of different deployment variants with tradeoffs and a significant complexity increase generation on generation. The supply chain gets reworked for end datacenter deployers, clouds, server OEMs / ODMs, and downstream component supply chains.
Today we are going to go from A to Z on the different form factors of GB200 and how they changed versus the prior 8 GPU HGX baseboard servers. We will break downs on unit volumes, supplier market share and cost for over 50 different subcomponents of the GB200 rack. Furthermore, we will dive into the hyperscale customization that changes the subcomponent supply chain heavily. Lastly we will also do a deep dive into the various types of liquid cooling architectures, deployment complexities, and the supply chain there.
Table of Contents:
GB200 form factors
Power budget
Compute tray architecture
Networking fabrics
NVLink fabric
NVL72
NVL36x2
Backend fabric (Infiniband/Ethernet)
Frontend fabric
Networking dollar content summary
Optics
Hyperscaler customization
Rack architecture changes & content
Heat transfer flow
L2A (Liquid to Air) vs L2L (Liquid to Liquid)
Redesigning data center infrastructure
Supply chain procurement decision maker & analysis
Liquid cooling components competition
Power delivery network, PDB, Busbar, VRM
Mechanical components
OEM / ODM Mapping
The 4 Rack Scale Form Factors of Blackwell
With GB200 Racks, there are 4 different major form factors offered, with customization within each.
GB200 NVL36x2
GB200 NVL36x2 (Ariel)
The first one is the GB200 NVL72 form factor. This form factor requires approximately 120kW per rack. To put this density into context, a general-purpose CPU rack supports up to 12kW/rack, while the higher-density H100 air-cooled racks typically only support about 40kW/rack. Moving well past 40kW per rack is the primary reason why liquid cooling is required for GB200.
Atomic Claim 1/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0001
Claim: GB200 NVL72 每個 rack 約需 120kW。
Frame:ATTRIBUTE· Mode:ESTIMATED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 2/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0002
Claim: GB200 因 rack 功率密度超過約 40kW 而需要 liquid cooling。
Frame:RELATION· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Source: SemiAnalysis
The GB200 NVL72 racks consists of 18 1U compute trays and 9 NVSwitch trays. Each compute tray is 1U in height and contains 2 Bianca boards. Each Bianca board is 1 Grace CPU and 2 Blackwell GPUs. The NVSwitch trays have two 28.8Tb/s NVSwitch5 ASICs.
Atomic Claim 3/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0003
Claim: GB200 NVL72 rack 由 18 個 1U compute trays 與 9 個 NVSwitch trays 組成。
Frame:NARY_RELATION· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 4/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0004
Claim: compute tray 高度為 1U，並包含 2 個 Bianca boards。
Frame:ATTRIBUTE· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 5/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0005
With the exception of one hyperscaler who plans to deploy this as the primary variant, we believe that this version will be rarely deployed until Blackwell Ultra as most datacenter infrastructure cannot support this high of a rack density even with direct-to-chip liquid cooling (DLC).
The next form factor is the GB200 NVL36 * 2 which is two racks side by side interconnected together. Most of the GB200 racks will use this form factor. ↗ Each rack contains 18 Grace CPUs and 36 Blackwell GPUs. Between the 2 racks, it still maintains being non-blocking all-to-all between all the 72 GPUs found in NVL72. Each compute tray is 2U in height and contains 2 Bianca boards. Each NVSwitch tray has two 28.8Tb/s NVSwitch5 ASIC chips. Each chip has 14.4Tb/s pointing backward toward the backplane and 14.4Tb/s pointing toward the front plate. Each NVswitch tray has 18 1.6T twin-port OSFP cages which connect horizontally to a pair NVL36 rack.
Atomic Claim 6/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0006
Atomic Claim 7/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0007
Atomic Claim 8/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0008
Claim: 在 GB200 NVL36x2 中，每個 compute tray 高度為 2U，並包含 2 個 Bianca boards。
Frame:ATTRIBUTE· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 9/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0009
Atomic Claim 10/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0010
Source: SemiAnalysis
The per rack power & cooling density is 66kW per rack for a total of 132kW for NVL36 racks * 2. This is the time to market solution as each rack is only 66kW/rack. Unfortunately, a NVL36x2 system does use ~10kW more power compared to NVL72 due to the additional NVSwitch ASICs and the requirement for cross rack interconnect cabling. NVL36x2 will have 36 NVSwitch5 ASICs in total compared to only 18 NVSwitch5 ASICs on the NVL72. Even with this increase in overall power of 10kW, most firms will deploy this version next year instead of NVL72 as their datacenter can’t support 120kW per rack density. ↗ We will discuss the reason for this later in the liquid cooling section.
Atomic Claim 11/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0011
The last form factor is the specific rack with a custom “Ariel” board instead of the standard Bianca. We believe this variant will primarily be used by Meta. Due to Meta’s recommendation system training and inferencing workloads, they require a higher CPU core and more memory per GPU ratio in order to store massive embedding tables and perform pre/post-processing on the CPUs.
The content is similar to the standard GB200 NVL72: but instead the Bianca board is swapped for an Ariel board that has 1 Grace CPU and 1 Blackwell GPU. Due to the doubling of Grace CPU content per GPU, this SKU will be more expensive even compared to NVL36x2. Similar to NVL36x2, each NVSwitch tray has 18 1.6T twin-port OSFP cages which connect horizontally to a pair NVL36 rack.
Source: SemiAnalysis
We believe that the majority of Meta’s allocation will be the normal NVL36x2 as that is more geared towards GenAI workloads while the Ariel version will just be for their largest Recommendation System workloads ↗. While there is nothing preventing Ariel from being used for GenAI workloads, the overprovisioning of CPUs means it will be inferior from a TCO standpoint ↗ due to higher capital cost and power.
Lastly, in Q2 2025, there will be a B200 NVL72 and NVL36x2 form factor that will use x86 CPUs instead of Nvidia’s in-house grace CPU. This form factor is called Miranda. We believe that the CPU to GPU per compute tray will stay the same at 2 CPUs and 4 GPUs per compute tray.
We believe that this variant of NVL72/NVL36x2 will have lower upfront capital cost compared to the Grace CPU version, with less revenue flowing to Nvidia. Since it is using an x86 CPU, there will be much lower CPU to GPU bandwidth compared to Grace C2C which can talk to the GPUs at up to 900GB/s bidirectional (450GB/s). Because of this TCO is questionable. Furthermore as the x86 CPUs will not be able to share power between the CPU and GPUs to optimize for the workload total peak power required is much higher. In our accelerator model, we have broken down which GB200 form factors & the exact volume each of the top 50 buyers will be deploying. ↗
Power Budget Estimates
We estimate that the max TDP of each compute tray is 6.3kW. Most of the power draw from the compute tray is from the two Bianca board and 8 fans in each tray. The NVSwitch tray in NVL72 does not need to connect between racks as such has a 170W lower power draw compared to NVL36. With NVL36, there are 18 1.6T ACC cables to connect horizontally to the neighboring rack. We will explain the NVLink topology in further sections. 123.6kW per NVL72 is the total power draw including the inefficiencies from rectifying from AC power from the whip to DC power that the compute tray takes in.
Atomic Claim 12/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0012
Claim: SemiAnalysis 估計每個 compute tray 的最大 TDP 為 6.3kW。
Frame:ATTRIBUTE· Mode:ESTIMATED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 13/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0013
Claim: compute tray 的主要功耗來自 2 個 Bianca boards 與每 tray 8 個 fans。
Frame:ATTRIBUTE· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 14/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0014
Atomic Claim 15/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0015
Atomic Claim 16/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0016
Claim: NVL72 的總功耗為 123.6kW，包含 AC 轉 DC 的整流損耗。
Frame:ATTRIBUTE· Mode:ESTIMATED· Mapping:PARTIAL
開啟逐條審核
Source: SemiAnalysis GB200 Component & Supply Chain Model ↗
While for NVL36*2, each rack has a max TDP of ~67kW while both rack pairs take in ~132kW. This is approximately 10kW more power draw compared to NVL72.
Atomic Claim 17/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0017
Compute Tray Diagrams & Cabling
The heart of the GB200 NVL72/NVL36x2 is the Bianca board. The Bianca board contains two Blackwell B200 GPUs and a single Grace CPU. The ratio between CPU and GPU is now 1:2 on a board compared to GH200, which is a 1:1 ratio. Most of the customers that evaluated GH200 have told Nvidia that it was too expensive as 1:1 CPU ratio was too much for their workloads. This is one of the main reasons why GH200 shipped in such low volumes compared to HGX H100 (2 x86 CPUs, 8 H100 GPUs). For Blackwell, GB200 volume is way up relatively and there will be crossover in unit shipments versus HGX Blackwell B100/B200. ↗
Atomic Claim 18/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0018
Claim: Bianca board 包含 2 顆 B200 GPUs 與 1 顆 Grace CPU。
Frame:NARY_RELATION· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 19/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0019
Source: SemiAnalysis
Normally in Hopper & Blackwell HGX servers, there are Broadcom PCIe switches between the CPU and the GPU. For the GB200, the CPU and GPU are both on the same PCB, reducing insertion loss to a point that there is no longer any need for switches or retimers between the CPU and GPU on the reference design. This is on the surface is extremely negative for Astera Labs. It now has ~35% short interest of free float, but that mostly from folks who aren’t following the supply chain deeply and only understand there are no retimers in the reference design. We will share more details below and in the GB200 Component & Supply Chain Model ↗.
Atomic Claim 20/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0020
Claim: 在 GB200 reference design 中，CPU 與 GPU 位於同一張 PCB，降低 insertion loss，因此 CPU 與 GPU 間不再需要 switches 或 retimers。
Frame:NARY_RELATION· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Another interesting thing about the reference design is that instead of using the typical MCIO PCIe x16 connectors to connect the main PCB board to a PCIe form factor ConnectX-7/8, the ConnectX-7/8 ICs now sit directly on top of the Bianca board using a mezzanine board via Mirror Mezz connectors.
Source: SemiAnalysis
This has the advantage of using the same cold plate to cool both the CPUs, GPUs, and ConnectX-7/8 NICs. The electrical lanes are routed to the OSFP cages at the front of the chassis with DensiLink connectors from the mezzanine board. This is similar to how Nvidia used DensiLink on their gold plated DGX H100 chassis to route from the ConnectX-7 to the OSFP cages.
Atomic Claim 21/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0021
Claim: GB200 reference design 使用同一個 cold plate 同時冷卻 CPU、GPU 與 ConnectX-7/8 NIC。
Frame:NARY_RELATION· Mode:ASSERTED· Mapping:PARTIAL
開啟逐條審核
Similar to the Dual GH200, within the same compute tray, there is a high speed Coherent NVLink connection that operates up to 600GB/s bidirectional bandwidth (300GB/s unidirectional). This is an extremely fast connection and allows the CPUs to share resources and memory similar to the HGX H100/B100/B200 servers which have 2 CPUs and have NUMA (Non-Uniform Memory Access) regions.
Atomic Claim 22/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0022
Claim: 同一個 compute tray 內的 coherent NVLink 連線最高提供 600GB/s 雙向頻寬（300GB/s 單向）。
Frame:ATTRIBUTE· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Source: Nvidia
Due to this coherent link that hooks up two Bianca board, you can share memory, storage & resources such as NICs between the CPUs. For this reason, you will be able to depopulate a frontend NICs and only have 1 frontend NIC per compute tray instead of 2 as suggested in the reference design. This is similar to how in x86, even though, you have 2 CPUs per server, you only need 1 frontend NIC since the CPUs are able to share resources. We will discuss this more in the frontend networking section.
Source: SemiAnalysis
In terms of how the 2700 Watts of power gets to the board, there are 4 RapidLock 12V DC and 4 RapidLock GND (Ground) Power Connectors located around the CPU and GPU’s respective voltage regulator modules (VRM). These 12V and GND power connectors will connect to the compute tray’s power distribution board (PDB). The power distribution board takes 48V DC from the rack level busbar and steps it down to 12V DC for the Bianca board. We will discuss the changes to the power delivery network for the system in the power delivery section later.
Source: SemiAnalysis
In terms of the internal compute tray cables + connectors, most of the cost is dominated by the Mirror mezz connectors that connect the ConnectX-7/8 mezzanine board to the Bianca board & by the DensiLink cables that connect from the ConnectX-7/8 to the OSFP cages on the front of the chassis.
Source: SemiAnalysis GB200 Component & Supply Chain Model ↗
In the Nvidia reference design, there are two Bluefield-3s per compute tray, but as explained in later sections, we believe that most firms will not opt for any Bluefield-3 at all. At the front of the chassis, you can find all your typical server-related management ports, like RJ45, USB, etc. There are also eight NVMe storage bays for local node-level storage, and you can also find your scale-out backend OSFP cages.
Source: SemiAnalysis
The backend cage leads us to discuss one of the most critical pieces of the GB200: networking.
Networking
Similar to the HGX H100, AMD MI300X, Intel Gaudi, AWS Trainium, there are 4 different networks in the GB200 systems:
Frontend Networking (Normal Ethernet)
Backend Networking (InfiniBand/RoCE Ethernet)
Accelerator Interconnect (NVLink)
Out of Band Networking
As a quick refresher, the frontend networking is just your normal ethernet network that you use to connect to the internet, SLURM/Kubernetes, networked storage, data loading, model checkpoints. This network is typically 25-50Gb/s per GPU, so on a HGX H100 server, it will be 200-400Gb/s per server, while on a GB200 computer tray node, it will be 200-800Gb/s per server depending on the configuration.
Atomic Claim 23/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0023
Claim: GB200 compute tray node 的 frontend networking 依配置約為 200–800Gb/s per server。
Frame:ATTRIBUTE· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Your backend network is used to scale out GPU-GPU communications across hundred to thousands of racks. This network could either be Nvidia’s Infiniband or Nvidia Spectrum-X Ethernet or Broadcom Ethernet. With the options from Nvidia being way more expensive compared to the Broadcom Ethernet solutions ↗.
Atomic Claim 24/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0024
Claim: GB200 類系統的 backend network 用於跨數百至數千個 racks 的 GPU-to-GPU scale-out，選項包括 Nvidia InfiniBand、Spectrum-X Ethernet 或 Broadcom Ethernet。
Frame:NARY_RELATION· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
The scale-up accelerator interconnect (NVLink on Nvidia ↗, Infinity Fabric/UALink on AMD, ICI on Google TPU ↗, NeuronLink on Amazon Trainium 2) is an ultra-high speed network that connects GPUs together within a system. On Hopper, this network connected 8 GPUs together at 450GB/s each while on Blackwell NVL72, it will connect 72 GPUs together at 900GB/ each. There is a variant of Blackwell called NVL576 that will connect 576 GPUs together but basically no customers will opt for it. In general, your accelerator interconnect is 8-10x faster than your backend networking.
Atomic Claim 25/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0025
Claim: Blackwell NVL72 的 scale-up accelerator interconnect 連接 72 顆 GPU、每顆約 900GB/s；Hopper 則是 8 顆 GPU、每顆 450GB/s。
Frame:COMPARISON· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 26/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0026
Claim: Accelerator scale-up interconnect 一般約比 backend networking 快 8–10 倍。
Frame:ATTRIBUTE· Mode:ASSERTED· Mapping:PARTIAL
開啟逐條審核
Lastly, there is your out of band management network which is used for re-imaging your operating system, monitor node health such as fan speed, temperatures, power draw, etc. Your baseboard management controller (BMC) on servers, PDUs, switches, CDUs are usually connected to this network to monitor and control these IT equipment.
NVLink Scale Up Interconnect
Compared to HGX H100, the frontend, backend and out of band networking of GB200 are mostly same with the exception of NVLink expanding outside of the chassis. Only hyperscaler customizations are different gen on gen. Previously in HGX H100, the 8 GPUs and 4 NVSwitch4 Switch ASICs are connected together using PCB traces as they are on the same PCB, the HGX baseboard.
Source: SemiAnalysis
Now on HGX Blackwell, the NVSwitch ASICs are in the middle to reduce the length of the PCB trace given the upgraded 224G SerDes
But on the GB200, the NVSwitches are on a different tray from GPUs and therefore you need to either use Optics or ACCs to connect between them.
Atomic Claim 27/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0027
Claim: GB200 的 NVSwitch 與 GPU 位於不同 tray，因此兩者之間需要 optics 或 ACC 連接。
Frame:NARY_RELATION· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Source: Nvidia
In NVL72, they keep the same flat 1 tier NVLink topology as HGX Hopper/Blackwell such that you can talk to any of the GPUs within the same rack with only 1 hop through an NVSwitch. This is unlike AMD & Intel’s current generation interconnect which connects directly from GPU to GPU without a switch, which leads to reduced Accelerator to Accelerator bandwidth.
Atomic Claim 28/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0028
Claim: NVL72 採 flat 1-tier NVLink topology，同 rack 任一 GPU 之間僅需經過 1 個 NVSwitch hop。
Frame:ATTRIBUTE· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Source: SemiAnalysis
In NVL36x2, it only takes 1 hop to get to any of the 36 GPUs within the same rack but in order to talk to the other 36 GPUs in the rack beside it, it takes 2 NVSwitch hops to get across the racks. Intuitively, one additional hop adds latency but is not noticeable for training. It will slightly impact inference, but not too much unless extremely high interactivity (>500TPS) at batch 1 without speculative decoding is the target. ↗ Note that is quite an unrealistic scenario we don’t expect anyone to utilize.
Atomic Claim 29/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0029
Claim: NVL36x2 中，同 rack 的 36 顆 GPU 需 1 hop；跨到相鄰 rack 的另 36 顆 GPU 則需 2 個 NVSwitch hops。
Frame:ATTRIBUTE· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Source: SemiAnalysis
Nvidia claims that if they used optics with transceivers, they would have needed to add 20kW per NVL72 rack. We did the math and calculated that it would need to use 648 1.6T twin port transceivers with each transceiver consuming approximately 30Watts so the math works out to be 19.4kW/rack which is basically the same as Nvidia’s claim. At about 550,800 per rack in just transceiver costs alone. When you mark that up by Nvidia’s 75% gross margin, that would mean $2,203,200 per rack of NVLink transceivers that the end customer would need to pay. This is one of the main reasons why DGX H100 NVL256 never shipped due to the massive cost of transceivers. ↗ Furthermore, bleeding edge transceivers like the 1.6T NVLink transceivers have way worse reliability compared to copper cables or even prior generation optics.
Atomic Claim 30/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0030
Claim: Nvidia 聲稱若 NVL72 使用 optical transceivers，將增加約 20kW/rack 的功耗。
Frame:ATTRIBUTE· Mode:ATTRIBUTED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 31/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0031
Claim: SemiAnalysis 計算 NVL72 若採 optics，需 648 個 1.6T twin-port transceivers、每個約 30W，合計約 19.4kW/rack。
Frame:ATTRIBUTE· Mode:ESTIMATED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 32/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0032
Claim: 以每個 1.6T transceiver 約 850 美元估算，NVL72 單 rack transceiver 成本約 550,800 美元。
Frame:ATTRIBUTE· Mode:ESTIMATED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 33/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0033
Claim: SemiAnalysis 以 Nvidia 75% gross margin 推算，NVL72 的 NVLink transceivers 終端價格約 2.2032 百萬美元/rack。
Frame:ATTRIBUTE· Mode:ESTIMATED· Mapping:COMPLETE
開啟逐條審核
As such, this is the reason Nvidia chose to use 5184 copper cables, which is a much cheaper, less power hungry, and more reliable option. Each GPU has 900GB/s unidirectional bandwidth. Each differential pair (DP) is capable of transmitting 200Gb/s in 1 direction thus it will take 72 DP per GPU for both directions. Since there are 72 GPUs per NVL72 rack, that would mean there is 5184 differential pairs. Each NVLink cable contains 1 differential pair thus there are 5184 cables.
Atomic Claim 34/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0034
Claim: Nvidia 在 NVL72 選擇使用 5,184 條 copper cables，以降低成本、功耗並提高可靠性。
Frame:NARY_RELATION· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 35/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0035
Claim: NVL72 每顆 GPU 為 900GB/s 單向頻寬，每 differential pair 為 200Gb/s 單向；雙向每 GPU 需 72 DP，72 顆 GPU 合計 5,184 DPs。
Frame:ATTRIBUTE· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
This is a massive increase in copper content generation on generation. In a bit of a whiplash, we have seen some investors estimate that there is around 216k/NVL72 rack but this is completely wrong.
Source: Coatue
First of all, how did people even get to ridiculous numbers such as 162. Since it takes 18 400Gb/s full duplex cables per GPU, that would bring the price per GPU to $3k. This figure is wrong by a massive margin.
Source: SemiAnalysis
Furthermore, there is a misconception that the cables are expensive. Most of the cost is not from the cables itself but instead in termination of the cables & the connectors. The connectors are expensive as they need to prevent crosstalk between different differential pairs. Crosstalk is extremely bad as it blurs out other signals and causes errors where the de-serializers can’t read the correct bits. Nvidia has chosen to use the Ultrapass Paladin backplane product from Amphenol for their NVLink backplane interconnect as the primary initial source.
Atomic Claim 36/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0036
Claim: Nvidia 的 NVLink backplane 初期主要採用 Amphenol Ultrapass Paladin backplane。
Frame:NARY_RELATION· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
We will use the primary source’s name for each connector and cable in the article, but the there are 3 sources with varying share over time, details of which we share in the full GB200 Component & Supply Chain Model ↗
Source: SemiAnalysis
Each Blackwell GPU is connected to an Amphenol Paladin HD 224G/s connector, each with 72 differential pairs. Then, that connector attaches to the backplane Paladin connector. Next, it will connect using SkewClear EXD Gen 2 Cable to the NVSwitch tray Paladin HD connector with 144 differential pairs per connector. From the NVSwitch Paladin connector to the NVSwitch ASIC chip, OverPass flyover cables are needed since there are 4 144 DP connectors (576 DPs) per switch tray and there would be way too much crosstalk to do PCB traces in such a small area. Furthermore loss over the PCB is worse than over the flyover cables.
Source: SemiAnalysis GB200 Component & Supply Chain Model ↗
With NVL36x2, each system will require an additional 162 1.6T twin-port horizontal ACC cables which are extremely pricey ↗ to connect the NVSwitch trays between Rack A and Rack B. We break down the ACC cable and chip market here. ↗ There are multiple players with significant share. Furthermore, an additional 324 DensiLink flyover cables will be required for the OSFP cages. These DensiLink flyover cables alone are more than $10,000 of additional cost per NVL36x2.
Atomic Claim 37/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0037
Claim: NVL36x2 每套系統額外需要 162 條 1.6T twin-port horizontal ACC cables，用於連接 Rack A/B 的 NVSwitch trays。
Frame:ATTRIBUTE· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 38/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0038
Claim: NVL36x2 額外需要 324 條 DensiLink flyover cables，單此項增加成本超過 10,000 美元。
Frame:ATTRIBUTE· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Source: SemiAnalysis
Moreover, it will require twice as many NVSwitch5 ASICs to enable the connection between Rack A and Rack B. This will bring the total NVLink copper cabling costs up by more than double versus NVL72.
Atomic Claim 39/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0039
Claim: NVL36x2 因跨 rack 連接需要約兩倍 NVSwitch5 ASIC，NVLink copper cabling 成本也超過 NVL72 的兩倍。
Frame:COMPARISON· Mode:ASSERTED· Mapping:PARTIAL
開啟逐條審核
Even though the NVLink backplane content is more than twice as expensive for NVL36x2 vs NVL72, most customers will opt for the NVL36x2 design due to power and cooling constraints which we will discuss below. To be clear while it is very pricey, both NVL36x2 and NVL72 have lower copper costs than the investor community thinks.
Source: SemiAnalysis GB200 Component & Supply Chain Model ↗
The real winner is the cabling vendors and active copper chip vendors in the supply chain who have a massive increase in volumes for leading edge 1.6T due to the prevalence of NVL36x2.
GB200 NVL576
As Jensen mentioned on stage, GB200 NVLink can connect to 576 Blackwell GPUs together. We believe this is done using a 2 tier fat tree topology with 18 planes. This is similar to what they planned for DGX H100 NVL256 to connect 16 NVL36 racks. It will use 288 L1 NVSwitch5 ASICs (144 1U switch trays) located in the compute racks like NVL36x2, and it will use 144 L2 NVSwitch ASICs (72 2U switch trays) located on dedicated NV Switch trays. Like NVL36x2, the connection between the GPUs and the L1 NVSwitch will use the same copper backplane since it is over a short distance.
Atomic Claim 40/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0040
Claim: GB200 NVLink 的 NVL576 方案可連接 576 顆 Blackwell GPUs。
Frame:ATTRIBUTE· Mode:ATTRIBUTED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 41/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0041
Claim: SemiAnalysis 判斷 NVL576 採 2-tier fat-tree topology、18 planes。
Frame:ATTRIBUTE· Mode:INFERRED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 42/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0042
Claim: SemiAnalysis 預估 NVL576 使用 288 顆 L1 NVSwitch5 ASIC（144 個 1U trays）與 144 顆 L2 NVSwitch ASIC（72 個 2U trays）。
Frame:ATTRIBUTE· Mode:EXPECTED· Mapping:PARTIAL
開啟逐條審核
Source: SemiAnalysis
Unfortunately, between the L1 NVSwitch and the L2 NVSwitch, the distance is greater than what copper can achieve; thus, optical connections must be used. Furthermore, the L2 NVSwitches use Flyover cables to go to the OSFP cages on the front of the chassis. The additional BOM cost for NVL576 is astronomical at over $5.6 million dollars (9.7k per GPU) that Nvidia would need to pay to their suppliers.
Atomic Claim 43/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0043
Claim: NVL576 的 L1 與 L2 NVSwitch 距離超出 copper 可達範圍，因此必須採 optical connections。
Frame:RELATION· Mode:ASSERTED· Mapping:COMPLETE
開啟逐條審核
Atomic Claim 44/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0044
Claim: SemiAnalysis 估計 NVL576 額外 BOM 成本超過 560 萬美元，約每 GPU 9.7k 美元。
Frame:ATTRIBUTE· Mode:ESTIMATED· Mapping:COMPLETE
開啟逐條審核
Applying a blanket 75% gross margin means customers would need to pay an additional 38.8k per GPU for NVL576 copper + optical connections. While Nvidia can cut back on margins, even at 0% for the scale out NVLink solution, it’s basically untenable. This is the exact same reason to why DGX H100 NVL256 never shipped due to the massive cost of transceivers ↗. Optics is far too expensive for the accelerator interconnect as the accelerator interconnect needs to be extremely high bandwidth.
Atomic Claim 45/45 · 2024-07-17_gb200-hardware-architecture-and-component::GB2-0045
Claim: 若假設 75% gross margin，SemiAnalysis 估計 NVL576 copper + optical connections 將增加約每 GPU 38.8k 美元的客戶成本。
Frame:ATTRIBUTE· Mode:ESTIMATED· Mapping:COMPLETE
開啟逐條審核
Source: SemiAnalysis GB200 Component & Supply Chain Model ↗
Backend Networking
The backend networking of GB200 is where most of the options crop up. Nvidia generally releases GPUs with the new generation of NIC and Switch ready to go, but this generation, due to Nvidia’s aggressive timelines, especially with 224G SerDes, the new networking comes halfway through the Blackwell generation. As such, all initial shipments of GB200 will utilize the same ConnectX-7 that shipped in the majority of H100 servers.
For the Backend Networking there a handful of different types of switches that customers will use depending on which NIC they utilize.
Quantum-2 QM9700 Infiniband NDR
Quantum-X800 QM3400 Infiniband XDR
Quantum-X800 QM3200 Infiniband NDR/XDR
Spectrum-X SN5600
Spectrum-X Ultra
In terms of the backend networking, the time to market shipments will all be the QM9700 Quantum-2 switch or Broadcom Tomahawk 5 just like H100 geneartion. Despite it being the same backend networking hardware, there is a big challenge with utilizing rail optimized designs ↗. This due to the port mismatch between the switch and how many ports there are on a rack. With NVL72, there are 4 GPUs per compute tray, which means in a 4 rail optimized design, each Quantum-2 switch should have 18 downlink ports.
Source: SemiAnalysis
Since each switch has the same number of uplink ports in a fat tree, that means only 36 out of the 64 ports will be used. In effect, each switch will have many idle ports. If 2 rails went to each switch, then that would be 72 ports which will be over what the QM9700 Quantum-2 switch offers. In order to utilize all ports within each Quantum-2 switch, there will be 9 non-rail optimized leaf switches for every 4 NVL72 rack.
Source: Nvidia
For ConnectX-7, you can also use the Q3200 Quantum-3 switch tray which contains 2 independent switches, each with 36 400Gb/s ports. This does not have a port mismatch and can use 4-rail optimized with 4 Q3200 Quantum-X800 Switch per NVL72.
Source: Nvidia
For the upgraded 800Gb/s ConnectX-8, which will ship starting Q2 2025 with Quantum-X800 Q3400 which has 144 800Gb/s ports distributed across 72 twin port OSFP ports. Since there is no port mismatch, most customers will choose the Nvidia recommended design of 4-rail optimized with the switch rack being end of rack (EoR).
With CX-8, you can also use Spectrum-X Ultra 800G, forgoing the costly and expensive Bluefield option that was required in the prior generation. We discuss the Quantum-X800 switch option here and how it will affect the optical transceiver market. ↗ Broadcom based Tomahawk 6 deployment variants will also arrive in the 2nd half of next year.
Source: SemiAnalysis
The transition from CX-7 to CX-8 will be the main impetus for the shift from 400G (4x100G) SR4 optical transceivers to 800G (4x200G) DR4 optical transceivers. With CX-7 on the GB200 NVL72, each GPU has 400G of bandwidth and is connected to one OSFP cage with the multimode 400G Single-Port SR4 transceiver ↗ with four optical lanes each powered by a multimode 100G VCSEL. With a CX-7 based network, the switch side usually employs the 800G twin-port SR8 or DR8 transceiver.
For CX-8, all the speeds double, 800G (4x200G) DR4 per GPU and 1.6T(8x200G) DR8 per OSFP cage on the switch end. Because development work on the 200G multimode VCSEL will not complete for another 9 to 18 months for the 1.6T ramp, the industry has instead turned to the single mode 200G EML instead.
Similar to DGX H100, Cedar-8 will be available where both CX-8 NIC ICs from each Bianca board goes into a single OSFP224 cage ↗**. ↗ **The advantages of requiring two 1.6T (8x200G lanes) twin port transceivers instead of four 800G(4x200G lanes) single port transceivers. Since a single port 4x200G transceiver is approximately 35% less expensive than an 8x200G twin port transceiver, by using Cedar-8 instead of two 4x200G transceivers, the cost will be 30% lower. Due to the cooling challenges of having 2x as much bandwidth in a single OSFP cage on the compute tray, we expect that most firms will not be using the Cedar-8.
Most firms will be sticking to ConnectX-7/ConnectX-8 at launch. Even firms such as Google, that have historically used custom backend NICs from the likes of Intel, will be switching back to the Nvidia ConnectX-8 NICs.
The only exception that will integrate their own backend NIC will be Amazon. We believe they will use their custom backend 400G (4x100G) NIC. This networking card will be different from their standard Nitro NICs as it will mostly be performance oriented.
Source: SemiAnalysis
In order to use custom backend NICs on the Bianca board instead of using ConnectX ICs on the mezzanine board, they will need to use an adapter mezzanine board that splits from the mirror mezz connectors into 8 MCIO PCIe connectors that go to the front of the chassis.
Source: HighYieldYT
Since there will be no ConnectX-7/8 or Bluefield-3, which both have integrated PCIe switches, a dedicated PCIe switch from Broadcom / Astera Labs will be required to connect the backend NICs to both the CPU and GPU. In SemiAnalysis GB200 Component & Supply Chain Model ↗, we breakdown the PCIe switch supplier, volume, and ASP. There are still hyperscaler custom design that includes PCIe Switches for between the CPU and NICs. Nvidia’s Miranda design is also quite different than GB200 on PCIe Lane Handling. Furthermore Amazon Trainium 2 deployments have large amounts of Astera Labs retimer content ↗.
Using a custom NIC brings extra engineering work to firms as they can’t use the default water cooling block which is engineered to cool the ConnectX ICs too. They would also need to run fresh thermal simulations to ensure that custom NICs in the front of the chassis have enough cooling capacity to not cause overheating problems. Moreover, they will not be able to use the 1U compute tray version used in NVL72.
They can only opt for the 2U NVL36 version which has enough air-cooling capacity at the front of the tray. All of this extra engineering work will delay the time to market for Amazon and anybody else attempting to use custom backend NICs. These challenges are why Google opted to use ConnectX-8 instead of continuing to use Intel’s IPUs for their GB200 servers.
Frontend Networking
In the reference design, there are two 400Gb/s Bluefield-3 per compute tray. Since there are 4 GPUs per compute tray, that would mean that each GPU gets 200Gb/s of frontend bandwidth. The most advanced HGX H100 server deployed today has a single 200-400Gb/s ConnectX-7 NIC for their frontend traffic. That is for 8 GPUs, meaning 25-50Gb/s per GPU. 200Gb/s of frontend bandwidth per GPU is an extreme amount and most customers will not opt for this additional cost. In general, the Nvidia reference design is overprovisioned for the absolute worst-case situation for them to sell you more content.
Source: SemiAnalysis
We believe that the only major customer that will use Bluefield-3 as a frontend NIC will be Oracle. They run a cloud service that needs frontend network virtualization but have deployed a custom NIC solution, unlike the other hyperscalers. Amazon, Google, & Microsoft all have custom frontend NICs that are present in all their general-purpose CPU servers and accelerated computing servers already. They intend to continue to use these solutions as they offer a great TCO advantage and are already vertically integrated into their networking/cloud software stack.
Ironically, the only firm (xAI) that has widely used Bluefield-3 for the AI cluster is not even using it for its intended DPU purpose. xAI is using their Bluefield-3 in NIC mode instead of DPU mode since first-generation Nvidia Spectrum-X Ethernet requires Bluefield-3 for backend NICs as a bandage solution. Spectrum-X800 Ultra will work with CX-8 backend NICs and will not require Bluefield-3/4s to operate properly.
Source: SemiAnalysis, Michael Dell ↗
Networking Cables + Transceivers Bill of Materials
Below, we have calculated the Bill of Material costs Nvidia pays their contract manufacturers. We will be only calculating the cost of the transceivers on the compute/NVSwitch tray end as the calculations get complex if you include switches, as clusters can be tier 2 or tier 3, or even 4 tiers in giant clusters.
Source: SemiAnalysis GB200 Component & Supply Chain Model ↗
You can see that with ConnectX-8, by using a Top of Rack design with DAC/ACC copper instead of a 4-rail optimized backend design; there are savings of ~$32k for just the backend network alone. Unfortunately, due to the tight power requirements of the compute racks, we believe that most people will have to put their backend switches in a different service rack and use optics to connect between them.
For the out of band management, this is all cheap copper RJ45 cables that cost less than a dollar each to connect from the compute/switch trays to the top of the rack out of band management switch. As mentioned above, the reference design is overkill for how much frontend NICs and bandwidth they have. We believe that most firms will have 200G of frontend bandwidth instead of having 2 BF-3 which is 800Gb/s of bandwidth in total per compute tray. This will lead to a saving of $3.5k per system just in transceiver costs alone.
Source: SemiAnalysis GB200 Component & Supply Chain Model ↗
On the optics and DSPs, Nvidia is expanding the supply chain greatly to Eoptolink from just Fabrinet and Innolight who were the vast majority. Eoptolink is focusing on 800G LPO and 1.6T with DSP.
The DSP game also changes massively. While Marvell was 100% share on Nvidia last generation with H100. This generation, Broadcom comes in a big way. We see both Innolight and Eoptolink looking to be adding Broadcom in volume for the DSP.
Furthermore, Nvidia has hired a number of DSP engineers and taped out a 1.6T DSP. We do not believe this will ramp in the near term, but if it does, it will be on the Fabrinet transceivers. The biggest challenge to ramping up the internal DSP is that Nvidia has used primarily the same long reach high power SerDes on each side of the DSP. Typically, DSPs have differently optimized SerDes on the optics facing vs NIC/Switch facing sides. Both sets of SerDes are optimized for power more than reach alone which was Nvidia’s main optimization point when they designed their 224G SerDes. Nvidia’s internal DSP guzzle too much power, and as such their in-house DSP is too difficult to ramp production of due to cooling challenges in the already toasty 1.6T transceivers. Nvidia’s DSP can also function as a retimer if needed, but the ACCs are enough.
We have market share and ASP of Optics provider and DSPs in the SemiAnalysis GB200 Component & Supply Chain Model ↗.
Get 20% off a group subscription ↗
The above only covered the high level and basics, but below, we will dive into all the subcomponents and BOM on the GB200 including substrate, PCB, CCL, Substrate, liquid cooling, Sidecars, CDUs, UQDs, Manifolds, Vapor Chambers, Cold Plates, BMCs, and Power Delivery below. We will also cover more on the hyperscale customization. We will also cover all the complexities and decision matrices for liquid cooling supply chain selections.
Substrate, PCB, and CCL
Due to the system architecture changes and higher IO / power density, the organic package substrate, CCL (copper clad laminate), glass fiber, and PCB all have big complexity increases from Blackwell HGX and GB200.
The substrate which the Blackwell GPU sits on is larger and also has 2 more layers. In the prior generation, Ibiden dominated with 100% share. This generation, Unimicron becomes a second source of ABF substrates to Ibiden for Blackwell GPU. Moreover, Unimicron is the main supplier of ABF for Grace CPU with Ibiden as second. Unimicron is moving up their capacity expansion dedicated to Nvidia dramatically from 1Q25 to 3Q24 due to big increases in GB200 demand vs B100/B200. Ibiden will still grow, but Unimicron is taking a lot of the slack up.
We should note there have been significant problems with CoWoS-L ramps for Nvidia ↗, and all the other issues people keep murmuring about are minor in comparison.
For the HGX architecture, there are two main boards, the SXM board, the UBB (universal baseboard). One GPU is attached to each SXM board, then 8 SXMs are placed onto one UBB.
Source: SemiAnalysis
For GB200 architecture, this completely changes as described above. The primary Bianca board is significantly more complex per GPU due to the addition of the CPU, collapse to higher density board with higher power / more IO density. Furthermore, there are a variety of complex daughterboards that demand high grade CCL (M7+) such as the mezzanine ConnectX-7/8. This is because it runs in switch mode connecting the CPU and GPU to each NIC. The NVSwitch board also requires high grade M7 CCL.
Source: SemiAnalysis
The copper clad laminate for Hopper SXM was sole source from EMC and they were the majority supplier on the UBB board as well. With Blackwell, they are losing share on Blackwell SXM and Bianca boards to Doosan. This is likely due to price competition from Doosan, in which EMC chooses not to engage with as EMC will achieve full utilization rates in 3Q24 with other high margin projects like ASICs and LEO Satellites. Lastly, NVSwitch board CCL is M7 grade HDI and is sole supplied by EMC.
Source: SemiAnalysis GB200 Component & Supply Chain Model ↗
Despite the big win, Doosan does not have enough capacity. We estimate nearly 1.9 million sheets a month are required to meet Nvidia’s demand based on our house view split between GB200 and HGX shipment for Blackwell. EMC has more than double of Doosan’s capacity at 4.3mn sheets/month in 3Q24. EMC planned to expand capacity in 2025 by 1.2mn sheets/month to meet the extra demand that Nvidia has and Doosan cannot fulfill. There will be a consistent trend of requiring higher grade CCL (M7+) and higher layer count PCBs that will drive PCB materials businesses stronger.
Doosan’s CCL business unit contributed only 4.1% of its revenue in 1Q24, but this will soar. Interestingly, Doosan is not only exposed to the AI theme through its CCL business. Doosan also has a subsidiary – Doosan Enerbility ↗ which has meme stock potential given its partnership with NuScale and supplies them with the small modular nuclear reactors ↗ which some want to power datacenters with ↗.
Glass fiber cloth is a key raw material of CCL. Currently, the glass fiber cloth market (NE grade) required for AI server application is dominated by Japanese suppliers, notably Nitto Boseki and Asahi Kasei. Nitto Boseki has about 60% market share of NE grade glass fiber cloth and is currently supplied constraint. The difference between Nitto Boseki and Asahi Kasei is the glass fiber yarn manufacturing capacity which is the upstream material to make glass fiber cloth. Nitto Boseki has glass fiber yarn capacity which they can use internally and sell externally. Naturally, Nitto Boseki will prioritize the glass fiber yarn supply for themselves. Asahi Kasei, on the other hand, probably needs to source from AGY who supplies the glass fiber yarn.
Keeping It Cool: Liquid Cooling Analysis
As thermal design power (TDP) increases from 700W for H100 to 1000W/1,200W for B200/GB200, the second biggest shift from Hopper to Blackwell hardware besides connectivity/networking is the adoption of direct-to-chip liquid cooling (DLC) to increase compute density on the rack level. This translates to an 7-20 times increase in cooling content per GPU compared to DGX H100 depending on the GB200 SKU as well as the split between liquid to air (L2A) and liquid to liquid (L2L) solutions.
Source: SemiAnalysis GB200 Component & Supply Chain Model ↗
Air cooling is widely adopted for H100 HGX servers using 3DVC (3D vapor chamber) and fans. However, this comes with a sacrifice, at up to 10.2kW TDP per server, sufficient rack space is required to accommodate 4 rack unit tall (1 rack unit = 1.75in) 3DVC to dissipate heat effectively. 8xH100 HGXX servers are generally around 5-8RU tall. Each rack usually has 2-4 H100 HGX nodes, so the TDP for each rack is around 20-40kW.
For Blackwell, the B100’s and B200’s TDP are at 700W and 1000W/1200W respectively, which requires even taller 3DVC or cold plates for direct liquid cooling (DLC). With the 2700W GB200 Bianca board, DLC is the only option. The chassis height per compute tray to drop to 1-2 RU for GB200, compared to the 9-10RU chassis height of an air-cooled B200 HGX server.
Besides the increase compute density, another motivation to adopt DLC is the gain on energy efficiency as total data center power lowers by more than 10.2% for DLC L2L solution. In this section we will talk about the implication of this architectural shift of cooling solution on DLC hardware supply chain, data center architecture, different DLC form factors, and CSP/end customers procurement dynamic.
Redesigning Rack Architecture for DLC
DLC requires more components than air cooling to deploy. While air cooling solution hardware consists of Thermal Interface Material (TIM), Integrated Heat Spreader (IHS) (100% supplied by Jentech), 3DVC, and fans. liquid cooling solutions replace the 3DVC and the fans with these following components:
Cold Plate - dissipates and spreads heat within a copper plate and transfers heat through a liquid loop.
Quick Disconnects (QD) - allow the connection and disconnection of fluid lines in a fast and convenient way, without loss of fluids.
Coolant Distribution Manifolds (CDM) - supplies cold coolant to each server and returns warm coolant back to the CDU.
Liquid to Air Coolant Distribution Units (L2A CDU), also known as sidecar, which includes:
Reservoir and Pump Unit (RPU), which includes:
Pump – pushes and circulates the coolant throughout the liquid cooling system (2N design for redundancy).
Water Tank (Reservoir) - facilitates easy filling and expansion to avoid pressure build up and helps in removing air bubbles.
Radiators - releases the heat absorbed into the air
Fans – blow the air away form the radiator
Liquid to Liquid Coolant Distribution Units (L2L CDU), which includes:
RPU
Brazed Plate Heat Exchanger (BPHE) - creates flowing channels between plates, with one fluid in odd number channels and the other in the even number channels, thus reaching the purpose of heat exchanging.
Source: SemiAnalysis GB200 Component & Supply Chain Model ↗
We estimated the liquid cooling components dollar content by the NVL72 and the NVL72 (36*2) system and by different CDU form factor. The graph shows that adopting L2L CDU is much more economical than adopting L2A form factor, given the much higher price per W ratio for L2A CDU.
Heat Transfer Flow from the Chip
For air cooling: Heat from the chip is conducted through the TIM (thermal interface material) to be evenly distributed among the area of the heat spreader.
Then, the heat goes to the 3DVC, where the heat enters the vapor chamber and travels up the heat pipe into the heat sink. Then, air flow from the fan carries the heat away from the heat sink away from the chassis.
Source: Nextron
For DLC solution: the heat transfer from IHS to the cold plate and the coolant inside the coldplate absorbs the heat and carries it away to the cooling distribution manifold (CDM). Then, the CDM collects all the coolant from each tray and it goes to the CDU where the coolant exchanges heat with the surrounding air/water through the radiator/BPHE depending on whether it is L2L or L2A. The cooled coolant returns to the CDM, where the coolant is distributed back to each tray. Finally, the coolant enters the cold plate again repeating the process.
Source: Boyd
Despite the Bianca board, including two B200 GPUs, one Grace CPU, and two Connect-X NICs, will be liquid cooled, the front half of the GB200 compute tray and NVSwitch tray, where the custom NICs/DPU, PDU, and the management board sit, are air-cooled. Hence, the compute tray will be ~85% liquid cooled and ~15% air-cooled.
Source: Asus, SemiAnalysis
Moreover, there will be a separate version of the cold plate for the Bianca board to support hyperscale custom NICs for backend networking as mentioned earlier. This is because custom NICs replaces the Connect-X NIC and move them to the front of the tray away from the Bianca board.
There has been a rumor floating around that there is a major heating issue on Bianca. This is mostly overblown. The issue is much smaller than described, and the solution has already been found.
Liquid Cooling Form Factors: L2A vs L2L
There are 3 main liquid cooling form factors:
DLC liquid to air (L2A)
DLC liquid to liquid (L2L)
The two DLC form factors will be ubiquitous for GB200 deployment. While immersion cooling technology exists, it is not mature enough for mass deployment, and lacks serviceability. Nvidia only recognizes air cooling and DLC as the approved cooling technologies, meaning customers don’t get warranty from Nvidia with immersion cooling. As for now, we have only heard about a few non-GB200 immersion cooling projects in progress in Taiwan, Singapore, and the Middle East (Omniva). These immersion cooling projects must use lower performance / TCO versions of Blackwell.
The architecture of the rack between L2A and L2L form factors are mostly similar. As explained before, each IT rack for both L2A and L2L includes cold plates, CDMs, and QDs. The cooling rack/ in-rack CDU is where L2A differs from L2L. In other words, L2A utilizes air as the medium to reject heat from the technology cooling system (TCS), while L2L utilizes facility water as the medium to reject heat from the TCS. TCS represents the cooling system between the IT equipment and the CDU.
L2A has two form factors. The first L2A form factor is the “Rear Door Heat Exchanger” (RDHx), which is an in-rack solution. The second L2A form factor involves a cooling rack, also known as the sidecar.
Source: SemiAnalysis
In both L2A systems, hot coolant passes through the radiator (heat exchanger), the heat dissipates into the ambient air and gets blown away by the fans, cooling down the coolant. The cooled coolant enters the reservoir and pump units (RPU) and goes back to the cold plate.
RDHx solution has the fans and the radiators attached to the back and the RPU placed at the bottom of the IT rack, while the sidecar solution has these components in a separate cooling rack. Essentially, the side car allows for more radiator space compared to the back of a rack. This reflects the difference between each cooling capacity. RDHx’s cooling capacity is around 30kW-40kw, while Sidecar CDU’s cooling capacity ranges from 70kW to 140kW. We don’t see many uses cases for RDHx as its cooling capacity is way too low to even handle the TDP of one NVL36 rack (66kw). On the other hand, sidecars are designed with capacity at 70kW and 140kW to solve for one NVL36 rack (66kW) and one NVL72 rack (120kW) respectively.
Source: GRCooling
Speaking of rear door heat exchanger (RDHx) liquid cooling solution, some may be confused with another form of air cooling oriented RDHx. From our check, half of the H100 100K cluster shipped to X.ai from Dell adopt this air-cooling enabled RDHx technology, which Elon claimed to be liquid cooling. In fact, Dell shipped what is known as the air to liquid (A2L) RDHx solution. Within the computer tray, it works the same as normal air cooling with 3DVC and fans.
Then, the heat is pulled through the chassis arriving at the heat exchanger (radiator coils) attached to the rear door. Chilled coolant from the CDU gets pumped through the heat exchanger and exchange heat with the hot air. Finally, the warm coolant carries part of the heat toward the CDU to reject heat from the TCS. Usually, the A2L RDHx alone does not have enough cooling capacity to handle all the heat. Hence, the rest of the heat will enter the ambience to be processed by datahall’s air conditioning (CRAC).
Source: Motivair
For L2L, there are two form factors, which are with in-rack CDU and in-row CDU. Unlike L2A, L2L coolant does not exchange heat through a radiator; instead, coolant enters the brazed plate heat exchanger (BPHE) in the CDU to exchange heat with facility water. After coolant returns to the RPU from the BPHE.
Source: SemiAnalysis
The difference between in-rack and in-row CDU is that in-rack CDU is placed at the bottom of each rack usually taking up ~4RU of rack space, while in-row CDU situates at the end of a rack row. The in-row CDU is connected to all the racks within the row through buried pipelines and has the cooling capacity to reject heat for all the racks. In-rack CDU typically has cooling capacity around 80kW, while in-row CDU has cooling capacity that ranges from 800kW to 2000kW.
Redesigning Data Center Infrastructure for DLC
L2A solution does not require redesigning of data center infrastructure, as heat is dumped into the data center air like traditional air-cooling. The traditional data hall air conditioner/handler (CRAC/CRAH) will remove the heat out of the data hall.
Source: Schneider
L2L solution, on the contrary, requires more site installation of piping to connect the server racks to the CDU and the CDU to the facility water system. Keep in mind that CRAC/CRAH are still required for L2L as only 85% of heat is removed by liquid, meaning that 15% of heat will enter the data center air. Total data center power consumption can decrease by more than 10.2% ↗ when using liquid cooling compared to 100% air-cooling.
Source: Schneider
Moreover, total usage effectiveness (TUE), a more insightful energy efficiency comparative metric between liquid and air cooling, decreases by more than 15%. ↗ The TUE for air cooling should be around 1.4 for hyperscalers (~1.55 for neoclouds), while the TUE drops to 1.15 for DLC L2L. With less fans required compared to air cooling and uninterrupted airflow into the data center air, the TUE for DLC L2A should still less than 1.4 of air cooling, which is much higher than 1.15 of DLC L2L as 100% of the heat must be removed by CRAC/CRAH.
It is obvious that L2L will be the long-term mainstream DLC solution given its energy efficiency and price per W advantage over L2A. However, L2L deployment faces 2 significant delaying factors and bottlenecks now. Firstly, CSPs are prioritizing speed of deployment over TUE gain. Secondly, permitting, which could take time to secure, is required to source water for the facility water system even though the site installation of piping to support L2L is already complete ↗. As a result, the hyperscalers are adopting L2A as the short-term contingency solution to ensure the quick deployment of the GB200 systems. We believe the majority of the GB200 volume will be in L2A form factor before 3Q25. ↗
Liquid Cooling Components Procurement Decision Chain
For liquid cooling components supplier, Nvidia only provides reference design partner for the IT rack listed on a reference vendor list (RVL). They do provide recommended vendors for the cooling rack (CDU), but they are not the same as reference design partners. ↗
Just like many other components, end customers are not obligated to use Nvidia reference design liquid cooling components. Even with their extremely capable R&D teams, some hyperscalers are relatively unfamiliar with liquid cooling supply chain. Therefore, ODMs or system integrators (SI) have more voice to recommend suppliers for liquid cooling components before the hyperscalers become familiar with the supply chain. There are two working models with unfamiliar liquid cooling components for hyperscalers:
ODM suggests suppliers to hyperscalers for qualification.
Hyperscalers authorize ODM to take responsibility and oversee qualification.
Source: SemiAnalysis
On the other hand, Tier 2 CSP and Neocloud may not have procurement team and R&D capability as adequate as the hyperscalers do. Especially, Neoclouds will rely on OEM like Supermicro, Dell, Lenovo, and HPE that offer no customization. They have two options:
Tier 2 CSP can procure from Nvidia’s reference design partners as they have already been qualified by Nvidia.
OEMs offer integrated solutions including DLC components that have been qualified and integrated by the OEMs.
Source: SemiAnalysis
Moreover, the procurement for cooling rack or CDU follows different processes to the IT rack components.
There are 3 working models for the sidecar (L2A CDU):
System integrator assembles + system integrator qualified components.
System integrator assembles + end customer qualified components.
Cooling solution suppliers integrated solution + ensuring sidecar integrates well with the IT rack.
L2L CDU are delivered fully integrated from supplier:
Cooling solution provider integrated solution (Vertiv, Motivair, etc.)
OEM integrated solution (Supermicro, etc)
System integrator integrated solution (Ingrasys, etc)
DLC Components Competitive Landscape Analysis
As the above analysis of the procurement decision indicates, being on the reference vendor list (RVL) doesn’t translate into orders for suppliers. Unless customers are purchasing DGX system, Nvidia doesn’t dictate on components procurement. Ultimately, the decision on suppliers is made by end customers or OEMs based on their procurement strategies. For example, Tier-1 CSPs aim to procure from 1 or 2 suppliers for each liquid cooling component.
At COMPUTEX 2024, over 80 liquid cooling suppliers showcased their liquid cooling solutions as well as production capacity. 5 of us from SemiAnalysis were there and had conversations with every firm.
With some simple math on capacity number, this raised an oversupply concerns before the party even got going. From our supply chain checks, many suppliers claim they have capacity, but very few have been listed on the reference vendor list let alone winning any significant orders. On top of this, quality and track record of the supplier is considered the most important determinant in the procurement decision chain. Hence, each component has varying competitive landscape and qualification entry barrier depending on different level of reliability requirements and customers procurement strategies.
Among the liquid cooling components, cold plate and QDs are where most of the leakage happens, so the hyperscalers are more likely to adopt solution from suppliers on the RVL. As end customers value quality and wish to avoid system failure over prices for components which are an insignificant percentage of GB200 BOM. We believe there will be less competition and pricing pressure on these components. In other words, there are very little incentives for end customers to replace incumbent suppliers at the risk of leakage. The supplier landscape of cold plate will be similar to that of 3DVC, but with some expansion: AVC, Cooler Masters, Delta, and Auras.
Suppliers of QD are mostly US/Europe based and the referenced vendors are currently CPC, Parker Hannifin, Danfoss, and Staubli. CPC had purposed built a factory with a clean room for the liquid cooling QDs. Besides the clean room requirement, the QDs have several mechanical IP, including latches and valves to ensure easy access and drip free servicing of the server. At the moment, there is supply tightness of QDs which could become a bottleneck of GB200 shipment.
CDM and CDU are the components that we believe will be more competitive than the cold plate and the quick disconnects. The most important part of the manufacturing process of the CDM is to solder the QDs onto the manifolds properly to prevent leakage. Soldering the QDs is not a difficult process hence the barrier to enter is low. As for the CDU, the components within a L2L CDU, BPHE, and RPU etc, are not difficult for any given supplier to make. However, we believe more established suppliers who provide complete integration of operation with telemetry/monitoring of the system and competent servicing/maintenance capacity will constitute a significant share of the L2L and L2A CDU market.
Power Delivery Network
As TDP per rack increases from 40kW for H100 4 nodes rack to 120kW GB200 NVL72 rack, power delivery network is centralized on the rack level. Conventionally, high voltage AC is stepped down and converted to 12VDC at node level power supply unit (PSU). For GB200, voltage step down AC to DC conversion happens at the rack level PSUs. The centralized design of GB200 increases power efficiency by 2% compared to conventional design with best-in-class grade PSU.
The gain in efficiency is due to the reduction in oversizing and redundancy factor (excess PSU capacity) on the rack level PSUs compared to node level PSUs. ↗ Then by replacing the 12VDC architecture with the 48VDC architecture sending higher voltage into the compute tray, the efficiency is increased by another percent.
Energizing AI: Power Delivery Competition Heats Up Vicor, MPS, Delta, ADI, Renesas, Infineon
Dylan Patel ↗, Myron Xie ↗, and 2 others
·
2023年8月1日
Read full story ↗
This is because of the shorter trace length leading to lower resistance loss as discussed in the negative Vicor pieces before. ↗ Note after we were negative, the stock fell to more than 50% in just a few months.
Source: APC, Schneider
Source: Schneider
Power shelves are symmetrically placed at the top and the bottom of the rack for GB200 rack.
Source: Eaton, SemiAnalysis
Each power shelf consists of 6 PSUs of 5.5kW, which equals to 33kW per power shelf. The power shelf accepts 346-480V AC from the power whip and outputs 48/50V DC, with a maximum current of 600A per power shelf.
Source: Eaton
GB200 NVL36 rack will have 2 power shelves per rack supplying 66kW of power, and NVL72 will have 4 power shelves supplying 132kW of power.
Source: Molex
Each power shelves will send power to the busbar at the rear of the rack running at 1200A. Power will enter the tray from the busbar bar connector at the back of the server tray, then it the cable will bring 48V DC to the power distribution board (PDB). The PDB will step down the DC to 12V and send it too all the board in the compute tray. The Bianca board receives power from the yellow 12V power connector placed at the edges. The fan receives power from the Bianca board through the 8 pin Molex fan connectors.
Source: Asus, SemiAnalysis
The supply chain for PDB and VRM on Bianca is quite diverse with even low quality companies such as Alpha Omega getting share on the PDB. The VRM on the Bianca board is much tighter, with the primary share players being Monolithic Power Systems, Renesas, and Infineon. We share more details on price and ASP in the GB200 Component & Supply Chain Model.
BMC
Baseboard management controller (BMC) is a specialized processor with logic control features that is used to remotely monitor and manage host systems. BMC can be accessed remotely via dedicated or shared networks and has multiple connections to the host system, allowing it to monitor hardware via sensors, flash BIOS/UEFI, provide host access via serial or physical/virtual KVM Console. It is able to power cycle and record host and log events. The BMC is used in servers and some datacenter equipment such as NIC, power supplier, DPU and so on.
Servers generally have an average of one to two BMCs per general server. For AI servers, the BMC content per server is different based on form factors and SKUs. In this section, we will analyze the BMC content for Nvidia AI servers as well as non-Nvidia AI servers.
In the GB200 NVL72 reference design with two Bluefield-3, there are 87 BMCs, including:
2 to 4 BMCs for compute tray
one BMC for HMC (hardware management console),
one BMC for DC-SCM (datacenter-ready security control module),
one BMC for each Bluefield-3 DPU,
one BMC for each NVSwitch tray,
one BMC for out of band management switch, and
one BMC for PDU/power shelves.
As explained in the previous networking section, very few CSPs and OEMs will do Bluefield-3 for the frontend network. If they adopt Bluefield-3, most would go with a single instead of the dual configuration for the frontend network. For NVL72 without Bluefield-3, there are only 51 BMCs (87 BMCs minus 36 BMCs for Bluefield-3).
Source: SemiAnalysis GB200 Component & Supply Chain Model ↗
Among major buyers, Google, Microsoft, Meta, and AWS will use their custom NICs instead of Bluefield-3 for HGX and NVL72/NVL36. Oracle uses Bluefield-3, but only one Bluefield-3 instead of two for its NVL36 * 2 system. As a result, there are 80 BMCs for Oracle’s NVL36 *2 server, less than 98 BMCs for a reference NVL36*2 form factor.
Source: SemiAnalysis GB200 Component & Supply Chain Model ↗
We can use BMC per GPU metric to compare the BMC content of different GB200 SKUs to that of HGX. Even the NVL72 system without Bluefield, which has the least BMC content among GB200 SKUs, has double the BMC to GPU ratio over HGX.
Although the BMC per GPU metric is sufficient to estimate the incremental BMC TAM of the GB200 system, we believe the BMC content increase per CPU metric provides another interesting perspective when analyzing BMC content. The GB200 system without Bluefield-3 have BMC content increase per CPU multiple around 1 times. In other words, BMC content does not increase per CPU from HGX H100 to the GB200 system without the significant incremental BMC content from Bluefield-3.
Source: SemiAnalysis GB200 Component & Supply Chain Model ↗
In conclusion, the main driver of higher BMC content for GB200 over HGX is the shift from the 2:8 CPU to GPU ratio for HGX to the 2:4 CPU to GPU ratio for GB200. This makes even more sense when you compare the BMC content increase per GPU against the BMC content increase per CPU of the Meta Ariel form factor with a 1:1 CPU to GPU ratio. (Typically, in AI accelerator and general servers, you have 2 CPUs per server chassis and the BMC content is directly tied to the server chassis. This is why it is essential to look at both per CPU and per GPU ratio).
We share estimates of the year-over-year change in BMC demands for general servers and AI servers based on the GPU/accelerator shipments in our Accelerator Model ↗and our GB200 Component & Supply Chain Model ↗. This includes the BMC breakdown for each client’s different AI form factors, as discussed above.
Key Assumptions for this estimation:
For HGX/NVL36/NVL72 servers for Google, Microsoft, AWS, Meta and Oracle, we calculate the BMC demand based on their system designs.
For HGX/NVL36/NVL72 for other buyers, we estimate usage of Bluefield-3, with our assumptions in the table below.
Source: SemiAnalysis GB200 Component & Supply Chain Model ↗
For other AI servers, the majority will have an average of 2.8 BMCs, the same as HGX.
For general servers we assume a 10% recovery in 2024 and 5% in 2025.
Given the assumption, our model estimates total BMC shipments increase in 2025, mainly driven by the 64% growth of BMC units for Nvidia-based AI servers. Other AI servers’ BMC demand should also grow. In terms of market size, the Nvidia NVL36/72 systems will increase BMC TAM in 2025, but our estimates differ from the street.
Source: SemiAnalysis GB200 Component & Supply Chain Model ↗
Consensus assumption of the capital market is that the majority or even all NVL36/72 servers will follow Nvidia’s reference design adopting Bluefield-3. However, this is incorrect as the major CSPs will have their custom NICs solutions instead of the Nvidia DPU. A sensitivity analysis must be done to see the upsides if other buyers (non-Google/Microsoft/Meta/AWS/Oracle) use more Bluefield-3. With 20% more buyers using Bluefield-3 based on Nvidia reference design, the BMC units would increase by 25k units, implying 0.5% upside to AI server BMC demand, or just 0.1% upside to total BMC market. If other buyers 100% follow Nvidia’s reference design, the upside to would be 11% for AI-related BMC volume, or only 2% for total BMC volume. In other word, even if other (non-Google/Microsoft/Meta/AWS/Oracle) buyer all adopt the Bluefield-3 DPU solution for their GB200 racks, the total BMC shipment would not increase significantly. This is because that BMC for AI servers only accounts for roughly 17% of BMC shipments only.
Source: SemiAnalysis GB200 Component & Supply Chain Model ↗
In an unrealistic scenario where all NVL36/72 buyers including Google/Microsoft/Meta/AWS/Oracle all follow Nvidia’s reference design in adopting Bluefield-3, they would require an additional 854k units of BMC, representing 18% upside to AI server BMC demand, or 3% upside to total BMC demand. This would drive the total BMC volume to grow 12% in 2025.
Source: SemiAnalysis GB200 Component & Supply Chain Model ↗
Mechanical Components
The two main mechanical components are the chassis and the rail kit. These both benefitted from significant content increases for HGX H100 given the higher 5U-8U chassis and considerable weight that comes with the 3DVC compared to a 1U-2U traditional server. Hence, the chassis and rail kit ASP for HGX both went up ~10 times compared to traditional server.
Source: Oracle
With the liquid cooling of GB200 architecture, high 5U-8U chassis and high spec rail kit are not necessary not suitable anymore. Hence, the rail kit content is downgraded from HGX to GB200, with a more similar spec to the traditional server. King Slide has an extremely solid technological and IP leadership over its competitors especially for high 5U-8U rail kit, in which King Slide has high share of HGX rail kit. Its 60% gross margin for lower spec 1U or 2U traditional and over 60% gross margin for 5U+ AI server rail kit reflects its pricing power. Note there is a content increase for 10U B200 liquid cooled servers.
Although Nan Juen(Repon) is still slightly behind King Slide in terms of IP for the 5U+ rail kit, it will gain share from King Slide at GB200 due to the lower IP barrier to overcome. However, this is unlikely to drive the pricing down as Nan Juen’s pricing is only slightly less than King Slide who determine the pricing for the market.
The chassis will have slightly better pricing than the traditional server of the same height due to better material and thicker chassis for GB200 compute tray. The main players of chassis for GB200 will be Chenbro and AVC. Note players like Ingrasys will make their own chassis.
OEM / ODM Mapping
OEM and ODM details are also very important as there is a big shift generation on generation. There is a fear from some that OEMs get much weaker. This is not the case because complexity of deployments is much higher. Another fear is that Quanta loses tons of share to Ingrasys, but that is only at Microsoft. We have all these details and more in the GB200 Component & Supply Chain Model.
Get 20% off a group subscription ↗
GB200 Component & Supply Chain Model ↗
Dylan Patel ↗
·
2024年7月16日
The SemiAnalysis GB200 Component & Supply Chain BoM Model details the components within a GB200 NVL72/NVL36 system as well as the different customized SKUs. We provide pricing and quantity of each component, which differs depending on the GB200 SKU. We also map out the supplier landscape for each component. The quantity of components and BoM costs for e…
Read full story ↗