SA260324R-0125
① SA Source
- Source: 開啟完整 SA 文章
- Section:
Speculative Decoding - Line hint:
137
Context Before
Speculative Decoding
A different way LPUs could improve decode phase latencies is by accelerating a speculative decoding setup, where we deploy draft models or Multi-Token Prediction (MTP) layers onto LPUs.
Evidence
For a decoding step of context N tokens, adding k additional tokens during forward pass (a warm prefill of k new tokens) marginally increases the latency when k << N. Using this property, speculative decoding uses a small draft model or MTP layers to predict k new tokens, saving time since small models have lower latency per decode step. To verify the draft tokens, the main model only needs one warm prefill of k new tokens, at the latency cost of roughly a single decode step. Speculative decoding usually boosts output token per decode step by 1.5 to 2 tokens, depending on the draft model / MTP accuracy. With its low latency capabilities, LPUs can further increase the latency savings and improve throughput.
Context After
Source: SemiAnalysis
② Atomic Claim
為了驗證 draft token，main model 只需要對 k 個新 token 做一次 warm prefill，其 latency cost 約等於一次 decode step。
- Epistemic Mode:
ASSERTED - Mapping Status:
PARTIAL
③ Semantic Frame
{
"comparison_expression": "一次 warm prefill（k 個新 token）的 latency cost 約等於一次 decode step。",
"entities": [
{
"id": "04_knowledge_base/Prefill",
"label": "prefill"
},
{
"id": "04_knowledge_base/Decode",
"label": "decode"
}
],
"frame_type": "COMPARISON",
"metric": "LATENCY_COST",
"operator": "APPROX_EQUAL",
"qualifiers": {
"condition_text": "用於驗證 draft token；draft token 與 main model 沒有 canonical entity",
"numeric_mentions": [
"1",
"k"
],
"temporal_mentions": []
}
}④ Canonical Entity Mapping
⑤ V2 Draft（需要修正時直接編輯）
Inline repair
只編輯以下三個 Draft 區塊。完成後將
review_action設為REQUEST_REPAIR，再手動執行 Repair Preview。
執行腳本前，Draft 不會寫入 Claim Store 或 Graph。
V2 Atomic Claim Draft
為了驗證 draft token，main model 只需要對 k 個新 token 做一次 warm prefill，其 latency cost 約等於一次 decode step。
V2 Semantic Frame Draft
{
"comparison_expression": "一次 warm prefill（k 個新 token）的 latency cost 約等於一次 decode step。",
"entities": [
{
"id": "04_knowledge_base/Prefill",
"label": "prefill"
},
{
"id": "04_knowledge_base/Decode",
"label": "decode"
}
],
"frame_type": "COMPARISON",
"metric": "LATENCY_COST",
"operator": "APPROX_EQUAL",
"qualifiers": {
"condition_text": "用於驗證 draft token；draft token 與 main model 沒有 canonical entity",
"numeric_mentions": [
"1",
"k"
],
"temporal_mentions": []
}
}V2 Revision Metadata Draft
{
"epistemic_mode": "ASSERTED",
"mapping_status": "PARTIAL"
}⑥ Human Review
請在 Properties 逐項確認：
- 原文 → Atomic Claim 是否忠實
- Atomic Claim → Semantic Frame 是否忠實
- Canonical Entity mapping 是否正確
- Epistemic mode 是否保留原文語氣
- 最後選擇
review_action
Review state
Markdown 內文不是正式 approval。只有 Apply bridge 寫入的 Decision Ledger event 才是正式決策。