clm_c6e14e642def478789f5a80d2bf01f04_r1

IX2-0243

① SA Source

• Source: 開啟完整 SA 文章
• Section: Nvidia Disagg Prefill and WideEP
• Line hint: 392

Context Before

For higher interactivity levels (low batch size), large scale-up world sizes tend not to deliver stronger performance. B300 disagg over IB has the same performance as GB300 with NVL72, since the workload is latency-bound, not bandwidth-bound. The massive NVLink bandwidth advantage of NVL72 doesn’t matter because not even the much slower IB link is saturated by the tiny batches of tokens in flight.

Prefill/decode disaggregation also plays a role. Prefill is compute-heavy and bursty; decode is memory-bandwidth-bound and steady-state. When they share the same GPUs, they interfere with each other, causing latency jitter and wasted capacity. Separating them onto dedicated GPU pools lets each run a workload matched to its characteristics, improving effective utilization. This is why disaggregated B200 configs outperform single-node B200 in the middle of the throughput-interactivity curve. PD separation combined with wider EP across more GPUs over IB amortizes weights more efficiently than cramming both phases onto a single 8-GPU node.

Evidence

Side Note: the 10x inference engineers at TogetherAI noticed an pattern for multi-turn traffic where the requirements of first turn prefill is much different from the following turns prefill’s and disaggregrated it leading to better TTFT performance

Context After

Source: SemiAnalysis InferenceX ↗

② Atomic Claim

補充：TogetherAI 的 inference engineers 發現 multi-turn traffic 中，第一輪 prefill 與後續輪次 prefill 的需求差異很大，因此將其進一步 disaggregate 後，可改善 TTFT。

• Epistemic Mode: ASSERTED
• Mapping Status: COMPLETE

③ Semantic Frame

{
  "attribute": "LATENCY",
  "context_nodes": [
    {
      "id": "04_knowledge_base/Time to First Token",
      "label": "TTFT"
    }
  ],
  "entity": {
    "id": "04_knowledge_base/Prefill",
    "label": "prefill"
  },
  "frame_type": "ATTRIBUTE",
  "qualifiers": {
    "condition_text": null,
    "numeric_mentions": [],
    "temporal_mentions": []
  },
  "value": {
    "numeric_mentions": [],
    "value_text": "補充：TogetherAI 的 inference engineers 發現 multi-turn traffic 中，第一輪 prefill 與後續輪次 prefill 的需求差異很大，因此將其進一步 disaggregate 後，可改善 TTFT。"
  }
}

④ Canonical Entity Mapping

Role	Surface Label	Canonical Target
entity	prefill	Prefill
context_0	TTFT	04_knowledge_base/Time to First Token

⑤ Human Review

請在 Properties 逐項確認：

• 原文 → Atomic Claim 是否忠實
• Atomic Claim → Semantic Frame 是否忠實
• Canonical Entity mapping 是否正確
• Epistemic mode 是否保留原文語氣
• 最後選擇 review_action

Review state

Markdown 內文不是正式 approval。只有 Apply bridge 寫入的 Decision Ledger event 才是正式決策。