clm_a55714530b1e413aa2a50e1da5baab13_r1

IX2-0392

① SA Source

• Source: 開啟完整 SA 文章
• Section: WideEP
• Line hint: 632

Context Before

WideEP

By now, most frontier AI labs employ Mixture of Experts (MoE) model architectures as opposed to dense. In MoE architectures, only a subset of “experts” are activated for each token. For instance, DeepSeek R1 has 671B total parameters, but only 37B active parameters. Specifically, DeepSeek R1 has 256 routed experts (and 1 shared expert) with each token being routed to 8 distinct experts. This architecture lends itself naturally to expert parallelism (EP), which evenly distributes expert weights across some number of GPUs.

Evidence

With TP=8, each expert’s weights are sharded across all 8 GPUs, meaning every expert activation requires an all-reduce across all GPUs & the reduction dims of the GEMM is smaller leading to lower arithmetic intensity, even though only 8 of 256 experts activate per token

Context After

Expert parallelism takes a more well-suited approach, assigning whole experts to individual GPUs. With EP=8, we divide the 256 experts per layer across 8 GPUs for a total of 32 experts/layer/GPU. Each GPU holds approximately 1/8th of the expert weights plus a full replica of the non-expert weights (attention projections, embeddings, normalization, and the shared expert). Since roughly 90%+ of DeepSeek R1’s parameters are routed expert weights, EP captures most of the memory savings, and replicating the remaining less than 30B non-expert parameters across all 8 GPUs is affordable.

The forward pass proceeds in two phases per layer. During attention, each GPU acts as an independent data-parallel rank, processing its own subset of requests using its replicated non-expert weights, no inter-GPU communication is needed. During the MoE phase, a lightweight router determines which experts each token requires, and tokens are dispatched to the appropriate GPUs via all-to-all communication. Each GPU executes its local experts on only the tokens routed to it, and results are returned via a second all-to-all.

② Atomic Claim

在 TP=8 下，每個 expert weights 都分散在 8 顆 GPUs，因此每次 expert activation 都需要跨所有 GPUs 執行 all-reduce；同時 GEMM reduction dimension 變小，降低 arithmetic intensity，即使每 token 實際只啟動 256 個 experts 中的 8 個。

• Epistemic Mode: ASSERTED
• Mapping Status: COMPLETE

③ Semantic Frame

{
  "attribute": "COUNT",
  "context_nodes": [
    {
      "id": "04_knowledge_base/All-Reduce",
      "label": "all-reduce"
    },
    {
      "id": "04_knowledge_base/Arithmetic intensity",
      "label": "arithmetic intensity"
    }
  ],
  "entity": {
    "id": "04_knowledge_base/GPU",
    "label": "GPUs"
  },
  "frame_type": "ATTRIBUTE",
  "qualifiers": {
    "condition_text": null,
    "numeric_mentions": [
      "8",
      "256"
    ],
    "temporal_mentions": []
  },
  "value": {
    "numeric_mentions": [
      "8",
      "256"
    ],
    "value_text": "在 TP=8 下，每個 expert weights 都分散在 8 顆 GPUs，因此每次 expert activation 都需要跨所有 GPUs 執行 all-reduce；同時 GEMM reduction dimension 變小，降低 arithmetic intensity，即使每 token 實際只啟動 256 個 experts 中的 8 個。"
  }
}

④ Canonical Entity Mapping

Role	Surface Label	Canonical Target
entity	GPUs	GPU
context_0	all-reduce	All-Reduce
context_1	arithmetic intensity	04_knowledge_base/Arithmetic intensity

⑤ Human Review

請在 Properties 逐項確認：

• 原文 → Atomic Claim 是否忠實
• Atomic Claim → Semantic Frame 是否忠實
• Canonical Entity mapping 是否正確
• Epistemic mode 是否保留原文語氣
• 最後選擇 review_action

Review state

Markdown 內文不是正式 approval。只有 Apply bridge 寫入的 Decision Ledger event 才是正式決策。