IX2-0068
① SA Source
- Source: 開啟完整 SA 文章
- Section:
Interactivity vs Throughput Tradeoff - Line hint:
99
Context Before
Source: SemiAnalysis
Most benchmark results we will show in this article are InferenceX is a curve. It is important to analyze throughput at various levels of interactivity/latency instead of just looking at maximum achieved throughput (which normally can only be achieved at a single low interactivity). With inference, there is no one size fits all use case. The level of interactivity and throughput needed depends on the use case. For instance, real-time speech models require extremely low latency so that the end user can maintain a natural “conversation” with the LLM, whereas a basic QA chatbot may allow for higher latency. We leave it up to the reader to look at the curve and apply this principle to identify where their use case falls on the throughput-interactivity curve.
Evidence
The Cost/Perf per TCO vs Interactivity/End-to-End Latency curve mostly follows the Throughput vs Interactivity/End-to-End Latency Curve: More tokens/hour leads to a lower cost per token as fixed $/hour costs are amortized over more tokens produced
Context After
Prefill and Decode Phases
Inference contains two main phases: prefill and decode. Prefill occurs during the first forward pass of a request’s lifetime. It is computationally intensive since all tokens in the request are processed in parallel. This phase is responsible for “filling up” the KV cache for a sequence. After prefill, responses are generated (or decoded) one token at a time. Each forward pass loads the entire KV cache for a sequence from HBM, while only performing the computation for a single token, making decode memory (bandwidth) intensive.
② Atomic Claim
Cost/Perf per TCO vs Interactivity／End-to-End Latency 曲線大致跟隨 Throughput vs Interactivity／End-to-End Latency 曲線：每小時產生的 token 越多，固定的每小時成本就能攤在更多 token 上，因此每 token 成本更低。
- Epistemic Mode:
ASSERTED - Mapping Status:
CLAIM_ONLY
③ Semantic Frame
{
"frame_type": "CLAIM_ONLY",
"qualifiers": {
"condition_text": null,
"numeric_mentions": [],
"temporal_mentions": []
},
"reason": "CAUSAL_ECONOMIC_RELATION_WITH_PARTIAL_CANONICALIZATION",
"statement": "Cost/Perf per TCO vs Interactivity／[[04_knowledge_base/E2E latency|End-to-End Latency]] 曲線大致跟隨 Throughput vs Interactivity／[[04_knowledge_base/E2E latency|End-to-End Latency]] 曲線：每小時產生的 token 越多，固定的每小時成本就能攤在更多 token 上，因此每 token 成本更低。"
}④ Canonical Entity Mapping
- 此
CLAIM_ONLYrevision 沒有 canonical Entity mapping。
⑤ Human Review
請在 Properties 逐項確認：
- 原文 → Atomic Claim 是否忠實
- Atomic Claim → Semantic Frame 是否忠實
- Canonical Entity mapping 是否正確
- Epistemic mode 是否保留原文語氣
- 最後選擇
review_action
Review state
Markdown 內文不是正式 approval。只有 Apply bridge 寫入的 Decision Ledger event 才是正式決策。