mlsanityData to 14 Oct 2026

LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale

arXiv 2408.05499 ↗First posted 10 Aug 2024cs.DC cs.AIcited by 31

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Recently, there has been an extensive research effort in building efficient large language model (LLM) inference serving systems. These efforts not only include innovations in the algorithm and software domains but also constitute developments of various hardware acceleration techniques. Nevertheless, there is a lack of simulation infrastructure capable of accurately modeling versatile hardware-software behaviors in LLM serving systems without extensively extending the simulation time. This paper aims to develop an effective simulation tool, called LLMServingSim, to support future research in LLM serving systems. In designing LLMServingSim, we focus on two limitations of existing simulators: (1) they lack consideration of the dynamic workload variations of LLM inference serving due to its autoregressive nature, and (2) they incur repetitive simulations without leveraging algorithmic redundancies in LLMs. To address these limitations, LLMServingSim simulates the LLM serving in the granularity of iterations, leveraging the computation redundancies across decoder blocks and reusing the simulation results from previous iterations. Additionally, LLMServingSim provides a flexible framework that allows users to plug in any accelerator compiler-and-simulation stacks for exploring various system designs with heterogeneous processors. Our experiments demonstrate that LLMServingSim produces simulation results closely following the performance behaviors of real GPU-based LLM serving system with less than 14.7% error rate, while offering 91.5x faster simulation speed compared to existing accelerator simulators.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 57 of its 84 references that are in the map. First posted on arXiv 10 Aug 2024.

Builds on

57 of 84 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  2. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov
  3. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug
  4. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  12. 2023A Survey of Large Language Modelscited by 2,576
38 more
  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  4. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  5. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  6. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  7. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  8. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  9. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  10. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  11. 2023A Comprehensive Overview of Large Language Modelscited by 648
  12. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  13. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  14. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  15. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  16. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  17. 2023Active Retrieval Augmented Generationcited by 516
  18. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 346
  19. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  20. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
  21. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  22. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  23. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
  24. 2023Retrieval meets Long Context Large Language Modelscited by 115
  25. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  26. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 106
  27. 2020A3: Accelerating Attention Mechanisms in Neural Networks with Approximationcited by 101
  28. 2020EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inferencecited by 70
  29. 2019Gemmini: Enabling Systematic Deep-Learning Architecture Evaluation via Full-Stack Integrationcited by 63
  30. 2024NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencingcited by 52
  31. 2022DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generationcited by 48
  32. 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
  33. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  34. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 45
  35. 2020Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformercited by 43
  36. 2023vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Trainingcited by 23
  37. 2024Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real Systemcited by 15
  38. 2023Pathfinding Future PIM Architectures by Demystifying a Commercial PIM Technologycited by 14

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 19.

31 papers.

Sort:

By area:

  1. 2026How Fast Can I Run My VLA? Demystifying VLA Inference Performance with VLA-Perfcited by 19
  2. 2024APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Servingcited by 15
  3. 2026LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructurecited by 9
  4. 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Servingcited by 7
  5. 2025Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modelingcited by 7
  6. 2025TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systemscited by 6
  7. 2025LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving Techniques in LLM Infrastructurecited by 6
  8. 2025DSD: A Distributed Speculative Decoding Solution for Edge-Cloud Agile Large Model Servingcited by 5
  9. 2026Revati: Transparent GPU-Free Time-Warp Emulation for LLM Servingcited by 5
  10. 2026Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulationcited by 3
  11. 2025MIST: A Co-Design Framework for Heterogeneous, Multi-Stage LLM Inferencecited by 2
  12. 2025Frontier: Simulating the Next Generation of LLM Inference Systemscited by 2
19 more
  1. 2025A Data-driven ML Approach for Maximizing Performance in LLM-Adapter Servingcited by 1
  2. 2025Scalable Synthesis of distributed LLM workloads through Symbolic Tensor Graphscited by 1
  3. 2026AgentServeSim: Serving-System Simulation and Policy Search for LLM Agent Programscited by 1
  4. 2026RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Servingcited by 1
  5. 2026Hydra: Phase-Aware Workload Characterization of LLM Inference across Edge SoC Generations, Backends, and Quantization Levelscited by 1
  6. 2025Photonic Fabric Platform for AI Acceleratorscited by 0
  7. 2025Systematic Exploration of Multi-core Architectures for Efficient LLM Serving using WaferAI-SIMcited by 0
  8. 2026Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Servingcited by 0
  9. 2026Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Servingcited by 0
  10. 2026Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulationcited by 0
  11. 2026EnergyLens: Predictive Energy-Aware Exploration for Multi-GPU LLM Inference Optimizationcited by 0
  12. 2026Charon: A Unified and Fine-Grained Simulator for Large-Scale LLM Training and Inferencecited by 0
  13. 2026ASTRA-sim 3.0: Next-Level Distributed Machine Learning Simulations via High-Fidelity GPU and Infrastructure Modelingcited by 0
  14. 2026Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attentioncited by 0
  15. 2026More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Servingcited by 0
  16. 2026Simthesizer: An Agent-Driven Simulation Framework for LLM Serving Systemscited by 0
  17. 2026PrefixBench-H100: Characterizing Prefix Reuse and Time-to-First-Token in H100 LLM Servingcited by 0
  18. 2026AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMscited by 0
  19. 2026Dissecting the Carbon-SLO Tension in Geo-Distributed LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LLM Inference Unveiled: Survey and Roofline Model Insights
  2. 2023Efficient LLM Inference on CPUs
  3. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models
  4. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  5. 2024New Solutions on LLM Acceleration, Optimization, and Application
  6. 2024Highly Optimized Kernels and Fine-Grained Codebooks for LLM Inference on Arm CPUs
  7. 2023Efficient LLM inference solution on Intel GPU
  8. 2025LLaMCAT: Optimizing Large Language Model Inference with Cache Arbitration and Throttling