mlsanityData to 7 Oct 2026

A Survey on Efficient Inference for Large Language Models

arXiv 2404.14294 ↗First posted 22 Apr 2024cs.CL cs.AIcited by 197

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large Language Models (LLMs) have attracted extensive attention due to their remarkable performance across various tasks. However, the substantial computational and memory requirements of LLM inference pose challenges for deployment in resource-constrained scenarios. Efforts within the field have been directed towards developing techniques aimed at enhancing the efficiency of LLM inference. This paper presents a comprehensive survey of the existing literature on efficient LLM inference. We start by analyzing the primary causes of the inefficient LLM inference, i.e., the large model size, the quadratic-complexity attention operation, and the auto-regressive decoding approach. Then, we introduce a comprehensive taxonomy that organizes the current literature into data-level, model-level, and system-level optimization. Moreover, the paper includes comparative experiments on representative methods within critical sub-fields to provide quantitative insights. Last but not least, we provide some knowledge summary and discuss future research directions.

Key facts. Cited by 197 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 248 of its 304 references that are in the map. First posted on arXiv 22 Apr 2024.

Builds on

248 of 304 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  2. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines18 Mar
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  4. 2024LLM Inference Unveiled: Survey and Roofline Model Insights26 Feb
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  6. 2024FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs8 Jan
  7. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache5 Jan
  8. 2023Fairness in Serving Large Language Models31 Dec
  9. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems23 Dec
  10. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov
  11. 2023FlashDecoding++: Faster Large Language Model Inference on GPUs2 Nov
  12. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug
1 more
  1. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 235

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  10. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2020Longformer: The Long-Document Transformercited by 3,362
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2023A Survey of Large Language Modelscited by 2,537
  11. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  13. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  14. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  15. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  16. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  17. 2024Mixtral of Expertscited by 1,785
  18. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  19. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  20. 2020Reformer: The Efficient Transformercited by 1,658
  21. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  22. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  23. 2020Rethinking Attention with Performerscited by 1,584
  24. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  25. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  26. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  27. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  28. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  29. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  30. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  31. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  32. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  33. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  34. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  35. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  36. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  37. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  38. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873

50 most cited of 235 All 235 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 57; 2025: 80; 2026: 60.

197 papers.

Sort:

By area:

  1. 2025Training Language Models to Reason Efficientlycited by 242
  2. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  3. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  4. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  5. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 122
  6. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
  7. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  8. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 108
  9. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 96
  10. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  11. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  12. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
38 more
  1. 2024Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decodingcited by 65
  2. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
  3. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 58
  4. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 51
  5. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  6. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  7. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  8. 2024EPIC: Efficient Position-Independent Caching for Serving Large Language Modelscited by 45
  9. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Modelscited by 45
  10. 2024Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengthscited by 44
  11. 2025SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruningcited by 39
  12. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains Morecited by 38
  13. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  14. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 35
  15. 2025Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latencycited by 32
  16. 2024LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Acceleratorscited by 31
  17. 2025R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routingcited by 28
  18. 2024Fast yet Safe: Early-Exiting with Risk Controlcited by 27
  19. 2025MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methodscited by 26
  20. 2024Quamba: A Post-Training Quantization Recipe for Selective State Space Modelscited by 25
  21. 2024CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaborationcited by 25
  22. 2025LaDiR: Latent Diffusion Enhances LLMs for Text Reasoningcited by 25
  23. 2025SpecEE: Accelerating Large Language Model Inference with Speculative Early Exitingcited by 23
  24. 2025Parallel Loop Transformer for Efficient Test-Time Computation Scalingcited by 23
  25. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  26. 2026FASTER: Rethinking Real-Time Flow VLAscited by 20
  27. 2024LOLA: LLM-Assisted Online Learning Algorithm for Content Experimentscited by 18
  28. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 18
  29. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inferencecited by 18
  30. 2026Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Taskscited by 18
  31. 2026TriAttention: Efficient Long Reasoning with Trigonometric KV Compressioncited by 18
  32. 2024AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structurescited by 17
  33. 2025Accelerating Large Language Model Reasoning via Speculative Searchcited by 16
  34. 2024Interactive Speculative Planning: Enhance Agent Efficiency through Co-design of System and User Interfacecited by 15
  35. 2025Growing a Multi-head Twig via Distillation and Reinforcement Learning to Accelerate Large Vision-Language Modelscited by 15
  36. 2025AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptationcited by 15
  37. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storagecited by 15
  38. 2025Scaling Law for Quantization-Aware Trainingcited by 15

50 most cited of 197 All 197 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques
  2. 2025Taming the Titans: A Survey of Efficient LLM Inference Serving
  3. 2024Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters
  4. 2025Leveraging the true depth of LLMs
  5. 2025Can abstract concepts from LLM improve SLM performance?
  6. 2023Efficient LLM Inference on CPUs
  7. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline
  8. 2023A Comprehensive Overview of Large Language Models