2023-07-10_gpt-4-architecture-infrastructure
Canonical Target Counts
| Target | Occurrences |
|---|
| 02_companies/OpenAI | 35 |
| 04_knowledge_base/GPT-4 | 27 |
| 04_knowledge_base/Batch size | 19 |
| 04_knowledge_base/Large language model | 14 |
| 04_knowledge_base/KV cache | 13 |
| 04_knowledge_base/Speculative Decoding | 11 |
| 04_knowledge_base/H100 | 10 |
| 04_knowledge_base/GPU | 9 |
| 04_knowledge_base/Mixture of Experts | 8 |
| 02_companies/GOOG | 7 |
| 02_companies/NVDA | 6 |
| 02_companies/META | 5 |
| 04_knowledge_base/Pre-training | 4 |
| 04_knowledge_base/GPT-3 | 4 |
| 04_knowledge_base/A100 | 4 |
| 04_knowledge_base/Pipeline parallelism | 3 |
| 04_knowledge_base/FLOP | 3 |
| 04_knowledge_base/Encoder | 3 |
| 04_knowledge_base/Decode | 3 |
| 04_knowledge_base/Chinchilla Scaling | 3 |
| 04_knowledge_base/Tensor Parallelism | 2 |
| 04_knowledge_base/Arithmetic intensity | 2 |
| 02_companies/ORCL | 2 |
| 04_knowledge_base/Fully Sharded Data Parallel | 2 |
| 04_knowledge_base/FP8 | 2 |
| 04_knowledge_base/Multi-Query Attention | 2 |
| 02_companies/Amazon | 2 |
| 04_knowledge_base/Prefill | 2 |
| 04_knowledge_base/FP16 | 2 |
| 04_knowledge_base/Head Node | 2 |
| 02_companies/Anthropic | 1 |
| 02_companies/DeepMind | 1 |
| 02_companies/ByteDance | 1 |
| 04_knowledge_base/Transformer architecture | 1 |
| 02_companies/Together AI | 1 |
| 04_knowledge_base/MI300 | 1 |
| 04_knowledge_base/1.6T | 1 |
| 02_companies/Scale AI | 1 |
| 04_knowledge_base/SpaceX Falcon launch vehicles | 1 |
| 02_companies/AMD | 1 |
| 02_companies/7735.T | 1 |
| 04_knowledge_base/NVLink | 1 |
| 02_companies/MSFT | 1 |
| 04_knowledge_base/Chip-to-Chip Interconnect | 1 |
| 04_knowledge_base/800G | 1 |
| 02_companies/Baidu | 1 |
| 04_knowledge_base/Conditional Routing | 1 |
| 04_knowledge_base/All-Reduce | 1 |
| 04_knowledge_base/Data Parallelism | 1 |
| 04_knowledge_base/Token-to-token latency | 1 |
| 04_knowledge_base/Sequence length | 1 |
| 04_knowledge_base/200G | 1 |
| 04_knowledge_base/400G | 1 |