IX2-0591
① SA Source
- Source: 開啟完整 SA 文章
- Section:
Migration to Multi Turn Real Multi-Turn Chat and Agentic Coding Datasets - Line hint:
810
Context Before
Currently, InferenceX uses completely random tokens as input for benchmarking. We then vary the ISL/OSL uniformly subject to the distribution [ISL*0.8, ISL], similarly for OSL. Because of the random data, we disable prefix caching in all our benchmarks, as the expected value of a prefix cache hit rate on completely random data is 0%. Furthermore, all the random data is single-turn, meaning each conversation contains only one prompt and one response. While this provides a good baseline Pareto frontier, it is not a practical benchmark setup that mimics real-world production inference workloads.
In the near term, we will create a basic multi-turn benchmark with a dataset like allenai/WildChat-4.8M ↗, which captures real users’ multi-turn conversations. In addition to enabling prefix caching on all scenarios, we will enable KV cache CPU offloading, as this is what we see being done in production workloads. This will more accurately evaluate the strengths and weaknesses of each chip. For instance, MI355X has 288GB HBM3e versus B200s 192GB. Therefore, we expect MI355X to perform better in a high concurrency multiturn scenarios as more memory can be allocated to the KV cache. On the other hand, in scenarios where the GPU KV cache is stressed and blocks are offloaded to the CPU, we expect the GBs to excel as these chips have 900GB/s bidirectional CPU-GPU bandwidth, compared to 128GB/s / 256GB/s on HGX with PCIe 5.0 and 6.0, respectively. Moreover, currently we see AMD’s software for CPU offloading is poor, which may negatively affect performance in the same scenarios.
Evidence
The point is: real-world multiturn datasets test more SOTA inference engine features and can capture more nuanced and robust performance data across all chips
Context After
With the rise of Claude Code, Codex, and Kimi, it is becoming increasingly important to benchmark performance in agentic coding scenarios. Like above, these scenarios are multi-turn but also include extremely long context conversations as well as tool use. In the next few months, we plan on creating a benchmark suite that will most accurately capture the performance of open models in these agentic coding scenarios across all chips.
Adding TPU, Trainium and More Models
② Atomic Claim
真實世界 multi-turn datasets 能測試更多 SOTA inference engine，並針對所有晶片取得更細緻且更具 robustness 的效能資料。
- Epistemic Mode:
ASSERTED - Mapping Status:
COMPLETE
③ Semantic Frame
{
"attribute": "LIFECYCLE_STATUS",
"context_nodes": [],
"entity": {
"id": "04_knowledge_base/Inference engine",
"label": "inference engine"
},
"frame_type": "ATTRIBUTE",
"qualifiers": {
"condition_text": null,
"numeric_mentions": [],
"temporal_mentions": []
},
"value": {
"numeric_mentions": [],
"value_text": "真實世界 multi-turn datasets 能測試更多 SOTA inference engine，並針對所有晶片取得更細緻且更具 robustness 的效能資料。"
}
}④ Canonical Entity Mapping
| Role | Surface Label | Canonical Target |
|---|---|---|
| entity | inference engine | 04_knowledge_base/Inference engine |
⑤ Human Review
請在 Properties 逐項確認：
- 原文 → Atomic Claim 是否忠實
- Atomic Claim → Semantic Frame 是否忠實
- Canonical Entity mapping 是否正確
- Epistemic mode 是否保留原文語氣
- 最後選擇
review_action
Review state
Markdown 內文不是正式 approval。只有 Apply bridge 寫入的 Decision Ledger event 才是正式決策。