mlsanityData to 14 Oct 2026

Taming the Titans: A Survey of Efficient LLM Inference Serving

arXiv 2504.19720 ↗First posted 28 Apr 2025cs.CL cs.AI cs.DCcited by 16

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large Language Models (LLMs) for Generative AI have achieved remarkable progress, evolving into sophisticated and versatile tools widely adopted across various domains and applications. However, the substantial memory overhead caused by their vast number of parameters, combined with the high computational demands of the attention mechanism, poses significant challenges in achieving low latency and high throughput for LLM inference services. Recent advancements, driven by groundbreaking research, have significantly accelerated progress in this field. This paper provides a comprehensive survey of these methods, covering fundamental instance-level approaches, in-depth cluster-level strategies, emerging scenario directions, and other miscellaneous but important areas. At the instance level, we review model placement, request scheduling, decoding length prediction, storage management, and the disaggregation paradigm. At the cluster level, we explore GPU cluster deployment, multi-instance load balancing, and cloud service solutions. For emerging scenarios, we organize the discussion around specific tasks, modules, and auxiliary methods. To ensure a holistic overview, we also highlight several niche yet critical areas. Finally, we outline potential research directions to further advance the field of LLM inference serving.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 89 of its 147 references that are in the map. First posted on arXiv 28 Apr 2025.

Builds on

89 of 147 references.

Closest ancestors · 24

Same direction, newest first

  1. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Information24 Apr
  2. 2025Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents10 Apr
  3. 2025FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling3 Apr
  4. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs2 Feb
  5. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  6. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scale15 Aug
  7. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  8. 2024Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving19 Jun
  9. 2024Queue management for slo-oriented large language model serving5 Jun
  10. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Serving8 May
  11. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention7 May
  12. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
12 more
  1. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction12 Apr
  2. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving2 Apr
  3. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines18 Mar
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  5. 2024LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization2 Mar
  6. 2024LLM Inference Unveiled: Survey and Roofline Model Insights26 Feb
  7. 2024InferCept: Efficient Intercept Support for Augmented Large Language Model Inference2 Feb
  8. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  9. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache5 Jan
  10. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems23 Dec
  11. 2023Efficient LLM inference solution on Intel GPU19 Dec
  12. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline22 May

Other references · 65

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  7. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  8. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  10. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  11. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  12. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
38 more
  1. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  2. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  3. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  4. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  5. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  6. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 269
  7. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  8. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
  9. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  10. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  11. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  12. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  13. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 117
  14. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
  15. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 112
  16. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  17. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  18. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
  19. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  20. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  21. 2024Learning How Hard to Think: Input-Adaptive Allocation of LM Computationcited by 70
  22. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structurecited by 67
  23. 2024Fast On-device LLM Inference with NPUscited by 62
  24. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programscited by 58
  25. 2023Striped Attention: Faster Ring Attention for Causal Transformerscited by 54
  26. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  27. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  28. 2023Distributed Inference and Fine-tuning of Large Language Models Over The Internetcited by 42
  29. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 37
  30. 2024Don't Stop Me Now: Embedding Based Scheduling for LLMscited by 35
  31. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  32. 2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 32
  33. 2023HexGen: Generative Inference of Large Language Model over Heterogeneous Environmentcited by 31
  34. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 31
  35. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 31
  36. 2024PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-designcited by 28
  37. 2024Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selectioncited by 25
  38. 2024PerLLM: Personalized Inference Scheduling with Edge-Cloud Collaboration for Diverse LLM Servicescited by 23

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 13.

16 papers.

Sort:

By area:

  1. 2026Predicting LLM Output Length via Entropy-Guided Representationscited by 9
  2. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 9
  3. 2025Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecturecited by 5
  4. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 4
  5. 2025Large Language Models for Crash Detection in Video: A Survey of Methods, Datasets, and Challengescited by 3
  6. 2026Few-Step Diffusion Language Models via Trajectory Self-Distillationcited by 3
  7. 2025Reasoning Language Model Inference Serving Unveiled: An Empirical Studycited by 1
  8. 2026VSPrefill: Vertical-Slash Sparse Attention with Lightweight Indexing for Long-Context Prefillingcited by 1
  9. 2026The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Projectcited by 1
  10. 2026SNLP: Layer-Parallel Inference via Structured Newton Correctionscited by 1
  11. 2026Panther: Faster and Cheaper Computations with Randomized Numerical Linear Algebracited by 0
  12. 2026Meeting SLOs, Slashing Hours: Automated Enterprise LLM Optimization with OptiKITcited by 0
4 more
  1. 2026Efficient Financial Language Understanding via Distillation with Synthetic Datacited by 0
  2. 2026Recency/Frequency Adaptive KV Caching for Large Language Model Servingcited by 0
  3. 2026Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practicecited by 0
  4. 2026CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks
  2. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency
  3. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective
  4. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models
  5. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques
  6. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
  7. 2023Fast Distributed Inference Serving for Large Language Models
  8. 2024A Survey on Efficient Inference for Large Language Models