mlsanityData to 7 Oct 2026

Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models

arXiv 2406.01698 ↗First posted 3 Jun 2024cs.AR cs.AI cs.DCcited by 12

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language models (LLMs) have shown remarkable performance across a wide range of applications, often outperforming human experts. However, deploying these gigantic models efficiently for diverse inference use cases requires carefully designed hardware platforms with ample computing, memory, and network resources. With constant innovation in LLM serving optimizations and model architecture evolving at breakneck speed, the hardware requirements to meet Service Level Objectives (SLOs) remain an open research question. To answer the question, we present an analytical tool, GenZ, to efficiently navigate the relationship between diverse LLM model architectures(Dense, GQA, MoE, Mamba), LLM serving optimizations(Chunking, Speculative decoding, quanitization), and AI platform design parameters. Our tool estimates LLM inference performance metrics for the given scenario. We have validated against real hardware platforms running various different LLM models, achieving a max geomean error of 5.82.We use GenZ to identify compute, memory capacity, memory bandwidth, network latency, and network bandwidth requirements across diverse LLM inference use cases. We also study diverse architectural choices in use today (inspired by LLM serving platforms from several vendors) to help inform computer architects designing next-generation AI hardware accelerators and platforms. The trends and insights derived from GenZ can guide AI engineers deploying LLMs as well as computer architects designing next-generation hardware accelerators and platforms. Ultimately, this work sheds light on the platform design considerations for unlocking the full potential of large language models across a spectrum of applications. The source code is available at https://github.com/abhibambhaniya/GenZ-LLM-Analyzer . Users can also be tried it on at https://genz-llm-analyzer.streamlit.app/ without any setup on your web browser.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 52 of its 116 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

52 of 116 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024LLM Inference Unveiled: Survey and Roofline Model Insights26 Feb
  2. 2023Efficient LLM inference solution on Intel GPU19 Dec
  3. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug

Other references · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
37 more
  1. 2024Mixtral of Expertscited by 1,785
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  4. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  5. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  6. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  7. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  8. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  9. 2024Large Language Models: A Surveycited by 506
  10. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  11. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  12. 2019Depth-Adaptive Transformercited by 227
  13. 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 178
  14. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 169
  15. 2023Speculative Decoding with Big Little Decodercited by 163
  16. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  17. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 125
  18. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 123
  19. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  20. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  21. 2022Broken Neural Scaling Lawscited by 102
  22. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decodingcited by 80
  23. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  24. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  25. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  26. 2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
  27. 2019Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirectcited by 41
  28. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  29. 2023vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Trainingcited by 22
  30. 2024Progressive Gradient Flow for Robust N:M Sparsity Training in Transformerscited by 17
  31. 2023MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systemscited by 14
  32. 2023A Hardware Evaluation Framework for Large Language Model Inferencecited by 14
  33. 2020Fast Stencil-Code Computation on a Wafer-Scale Processorcited by 10
  34. 2024Enabling Unstructured Sparse Acceleration on Structured Sparse Acceleratorscited by 8
  35. 2023Allies: Prompting Large Language Model with Beam Searchcited by 6
  36. 2022Mission Apollo: Landing Optical Circuit Switching at Datacenter Scalecited by 4
  37. 2023Chiplet Cloud: Building AI Supercomputers for Serving Large Generative Language Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 5.

12 papers.

  1. 2024TheoremLlama: Transforming General-Purpose LLMs into Lean4 Expertscited by 58
  2. 2024InfiniPot: Infinite Context Processing on Memory-Constrained LLMscited by 27
  3. 2026How Fast Can I Run My VLA? Demystifying VLA Inference Performance with VLA-Perfcited by 17
  4. 2025TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systemscited by 6
  5. 2025Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Expertscited by 4
  6. 2024Template-Driven LLM-Paraphrased Framework for Tabular Math Word Problem Generationcited by 3
  7. 2025MIST: A Co-Design Framework for Heterogeneous, Multi-Stage LLM Inferencecited by 2
  8. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environmentscited by 0
  9. 2026SMEPilot: Characterizing and Optimizing LLM Inference with Scalable Matrix Extensionscited by 0
  10. 2026Think Before You Grid-Search: Floor-First Triage for LLM Servingcited by 0
  11. 2026LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Servingcited by 0
  12. 2026SLIM: Saturation-Aware Lightweight Performance Modeling for LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks
  2. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
  3. 2025Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modeling
  4. 2025Taming the Titans: A Survey of Efficient LLM Inference Serving
  5. 2023Efficient LLM Inference on CPUs
  6. 2026RoofLang: Enabling AI-Driven Architecting of LLM Inference Systems
  7. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale
  8. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline