mlsanityData to 14 Oct 2026

Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems

arXiv 2312.15234 ↗First posted 23 Dec 2023cs.LG cs.AI cs.DCcited by 102

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

In the rapidly evolving landscape of artificial intelligence (AI), generative large language models (LLMs) stand at the forefront, revolutionizing how we interact with our data. However, the computational intensity and memory consumption of deploying these models present substantial challenges in terms of serving efficiency, particularly in scenarios demanding low latency and high throughput. This survey addresses the imperative need for efficient LLM serving methodologies from a machine learning system (MLSys) research perspective, standing at the crux of advanced AI innovations and practical system optimizations. We provide in-depth analysis, covering a spectrum of solutions, ranging from cutting-edge algorithmic modifications to groundbreaking changes in system designs. The survey aims to provide a comprehensive understanding of the current state and future directions in efficient LLM serving, offering valuable insights for researchers and practitioners in overcoming the barriers of effective LLM deployment, thereby reshaping the future of AI.

Key facts. Cited by 102 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 189 of its 337 references that are in the map. First posted on arXiv 23 Dec 2023.

Builds on

189 of 337 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memory12 Dec
  2. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov
  3. 2023FlashDecoding++: Faster Large Language Model Inference on GPUs2 Nov
  4. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug

Other references · 185

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
38 more
  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2023Mistral 7Bcited by 3,164
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  13. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  14. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  15. 2020Reformer: The Efficient Transformercited by 1,668
  16. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  17. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  18. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  19. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  20. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  21. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  22. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  23. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  24. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  25. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  26. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  27. 2018Hierarchical Neural Story Generationcited by 1,193
  28. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  29. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  30. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  31. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  32. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  33. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  34. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  35. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  36. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  37. 2019Are Sixteen Heads Really Better than One?cited by 816
  38. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762

50 most cited of 185 All 185 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 46; 2025: 28; 2026: 28.

102 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  2. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 298
  3. 2024Tool Learning with Large Language Models: A Surveycited by 223
  4. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  5. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  6. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  7. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  8. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  9. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  10. 2024Large Language Models and Games: A Survey and Roadmapcited by 93
  11. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 87
  12. 2024Frame-Voyager: Learning to Query Frames for Video Large Language Modelscited by 83
38 more
  1. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 78
  2. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  3. 2024Speculative Diffusion Decoding: Accelerating Language Generation through Diffusioncited by 57
  4. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 55
  5. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  6. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  7. 2025MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMscited by 43
  8. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 38
  9. 2025Optimizing Model Selection for Compound AI Systemscited by 36
  10. 2024OpenCity: A Scalable Platform to Simulate Urban Activities with Massive LLM Agentscited by 35
  11. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  12. 2024CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaborationcited by 25
  13. 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directionscited by 23
  14. 2025ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systemscited by 18
  15. 2024Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUscited by 17
  16. 2024Agreement-Based Cascading for Efficient Inferencecited by 16
  17. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  18. 2024Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generationcited by 15
  19. 2024APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Servingcited by 15
  20. 2025Seesaw: High-throughput LLM Inference via Model Re-shardingcited by 15
  21. 2025Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimationcited by 14
  22. 2024BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Modelscited by 13
  23. 2024Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decodingcited by 13
  24. 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inferencecited by 13
  25. 2024Efficient Streaming LLM for Speech Recognitioncited by 12
  26. 2025Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysiscited by 12
  27. 2024Efficient Inference for Large Language Model-based Generative Recommendationcited by 11
  28. 2024Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Modelscited by 10
  29. 2025PRISM: Privacy-Aware Routing for Adaptive Cloud-Edge LLM Inference via Semantic Sketch Collaborationcited by 10
  30. 2024Affordable Generative Agentscited by 9
  31. 2024Can LLMs get help from other LLMs without revealing private information?cited by 9
  32. 2024A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contextscited by 9
  33. 2025TPS-Bench: Evaluating AI Agents' Tool Planning & Scheduling Abilities in Compounding Taskscited by 9
  34. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 9
  35. 2026The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibilitycited by 8
  36. 2024Coarse-to-Fine Highlighting: Reducing Knowledge Hallucination in Large Language Modelscited by 7
  37. 2025AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelismcited by 7
  38. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Servingcited by 6

50 most cited of 102 All 102 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023An Automated Survey of Generative Artificial Intelligence: Large Language Models, Architectures, Protocols, and Applications
  2. 2024A critical review of methods and challenges in large language models
  3. 2025Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
  4. 2023A Comprehensive Overview of Large Language Models
  5. 2025From Large to Super-Tiny: End-to-End Optimization for Cost-Efficient LLMs
  6. 2026AE-LLM: Adaptive Efficiency Optimization for Large Language Models
  7. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques
  8. 2026Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks