mlsanityData to 7 Oct 2026

LLM Inference Serving: Survey of Recent Advances and Opportunities

arXiv 2407.12391 ↗First posted 17 Jul 2024cs.DC cs.AIcited by 46

Not on the map. Why?

Abstract (arXiv)

This survey offers a comprehensive overview of recent advancements in Large Language Model (LLM) serving systems, focusing on research since the year 2023. We specifically examine system-level enhancements that improve performance and efficiency without altering the core LLM decoding mechanisms. By selecting and reviewing high-quality papers from prestigious ML and system venues, we highlight key innovations and practical considerations for deploying and scaling LLMs in real-world production environments. This survey serves as a valuable resource for LLM practitioners seeking to stay abreast of the latest developments in this rapidly evolving field.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Builds on 51 of its 63 references that are in the map. First posted on arXiv 17 Jul 2024.

Builds on

51 of 63 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  5. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  6. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  7. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  8. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  9. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  11. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  12. 2022Efficiently Scaling Transformer Inferencecited by 529
38 more
  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  2. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  3. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  4. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  5. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  6. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  7. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  8. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  9. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  10. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  11. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  12. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  13. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  14. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  15. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  16. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 110
  17. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  18. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  19. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  20. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  21. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  22. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  23. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  24. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 83
  25. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  26. 2023Fairness in Serving Large Language Modelscited by 77
  27. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  28. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 69
  29. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
  30. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 44
  31. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 43
  32. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 36
  33. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 36
  34. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
  35. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inferencecited by 28
  36. 2024Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selectioncited by 25
  37. 2024PerLLM: Personalized Inference Scheduling with Edge-Cloud Collaboration for Diverse LLM Servicescited by 23
  38. 2024Toward Sustainable GenAI using Generation Directives for Carbon-Friendly Large Language Model Inferencecited by 13

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 22; 2026: 16.

46 papers.

Sort:

By area:

  1. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  2. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 76
  3. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  4. 2025Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inferencecited by 32
  5. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 28
  6. 2025Large Reasoning Models Learn Better Alignment from Flawed Thinkingcited by 13
  7. 2025Shape it Up! Restoring LLM Safety during Finetuningcited by 11
  8. 2025Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inferencecited by 6
  9. 2024The Impact of Hyperparameters on Large Language Model Inference Performance: An Evaluation of vLLM and HuggingFace Pipelinescited by 5
  10. 2025Labeling Free-text Data using Language Model Ensemblescited by 5
  11. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniquescited by 5
  12. 2025BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Servingcited by 5
34 more
  1. 2026Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inferencecited by 5
  2. 2025DeServe: Towards Affordable Offline LLM Inference via Decentralizationcited by 4
  3. 2025Unveiling the Landscape of LLM Deployment in the Wild: An Empirical Studycited by 4
  4. 2024Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Cachingcited by 3
  5. 2025Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inferencecited by 3
  6. 2025RAP: Runtime Adaptive Pruning for LLM Inferencecited by 3
  7. 2025Disaggregated Prefill and Decoding Inference System for Large Language Model Serving on Multi-Vendor GPUscited by 3
  8. 2024Watermarking Large Language Models and the Generated Content: Opportunities and Challengescited by 2
  9. 2024From Cool Demos to Production-Ready FMware: Core Challenges and a Technology Roadmapcited by 2
  10. 2025Integrating Personality into Digital Humans: A Review of LLM-Driven Approaches for Virtual Realitycited by 2
  11. 2025Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rankcited by 2
  12. 2025MatKV: Trading Compute for Flash Storage in LLM Inferencecited by 2
  13. 2026Distilling Lightweight Domain Experts from Large ML Models by Identifying Relevant Subspacescited by 2
  14. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestrationcited by 2
  15. 2026Characterizing Performance-Energy Trade-offs of Large Language Models in Multi-Request Workflowscited by 2
  16. 2024Less is More: Optimizing Function Calling for LLM Execution on Edge Devicescited by 1
  17. 2025Intelligent Orchestration of Distributed Large Foundation Model Inference at the Edgecited by 1
  18. 2025FlowSpec: Continuous Pipelined Speculative Decoding for Efficient Distributed LLM Inferencecited by 1
  19. 2025Reasoning Language Model Inference Serving Unveiled: An Empirical Studycited by 1
  20. 2025Federated Attention: A Distributed Paradigm for Collaborative LLM Inference over Edge Networkscited by 1
  21. 2025Joint Partitioning and Placement of Foundation Models for Real-Time Edge AIcited by 1
  22. 2026Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurationscited by 1
  23. 2024Reducing the Cost of Dropout in Flash-Attention by Hiding RNG with GEMMcited by 0
  24. 2025Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Servingcited by 0
  25. 20263D-Learning: Diffusion-Augmented Distributionally Robust Decision-Focused Learningcited by 0
  26. 2026Rethinking LLMOps for Fraud and AML: Building a Compliance-Grade LLM Serving Stackcited by 0
  27. 2026Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practicecited by 0
  28. 2026LCAi: Life Cycle Assessment with big data fusion and retrieval-augmented generation-assisted interpretationcited by 0
  29. 2026A-MADiff: Attention-Guided Multi-Agent DRL with Diffusion Policies for Memory-Aware Task Orchestration in Mobile AIGC Networkscited by 0
  30. 2026Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Servingcited by 0
  31. 2026End-to-End Latency-Minimizing and Load-Balanced Request Scheduling for Edge LLM Inference in Agentic AI Servicescited by 0
  32. 2026Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scalingcited by 0
  33. 2026Towards An LLM-Driven Unified Conversion Framework for BT and FSM in Autonomous Intelligent Systemscited by 0
  34. 2026Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testingcited by 0