mlsanityData to 13 Oct 2026

Efficient Large Language Models: A Survey

arXiv 2312.03863 ↗First posted 6 Dec 2023cs.CL cs.AIcited by 185

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable capabilities in important tasks such as natural language understanding and language generation, and thus have the potential to make a substantial impact on our society. Such capabilities, however, come with the considerable resources they demand, highlighting the strong need to develop effective techniques for addressing their efficiency challenges. In this survey, we provide a systematic and comprehensive review of efficient LLMs research. We organize the literature in a taxonomy consisting of three main categories, covering distinct yet interconnected efficient LLMs topics from model-centric, data-centric, and framework-centric perspective, respectively. We have also created a GitHub repository where we organize the papers featured in this survey at https://github.com/AIoT-MLSys-Lab/Efficient-LLMs-Survey. We will actively maintain the repository and incorporate new research as it emerges. We hope our survey can serve as a valuable resource to help researchers and practitioners gain a systematic understanding of efficient LLMs research and inspire them to contribute to this important and exciting field.

Key facts. Cited by 185 papers in the map (data through 13 Oct 2026). Builds on 263 of its 365 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

263 of 365 references.

References · 263

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  4. 2020Language Models are Few-Shot Learnerscited by 32,320
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  6. 2023GPT-4 Technical Reportcited by 18,278
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  10. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  11. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  12. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,502
38 more
  1. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,657
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,658
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,628
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,348
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,213
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,898
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,637
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
  11. 2020Longformer: The Long-Document Transformercited by 3,393
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,388
  13. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,181
  14. 2023Mistral 7Bcited by 3,123
  15. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,044
  16. 2022Scaling Instruction-Finetuned Language Modelscited by 3,007
  17. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,570
  18. 2023A Survey of Large Language Modelscited by 2,559
  19. 2022Emergent Abilities of Large Language Modelscited by 2,499
  20. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,440
  21. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,137
  22. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,994
  23. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,978
  24. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,851
  25. 2017Decoupled Weight Decay Regularizationcited by 1,788
  26. 2020Big Bird: Transformers for Longer Sequencescited by 1,681
  27. 2020Reformer: The Efficient Transformercited by 1,662
  28. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,625
  29. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,618
  30. 2020Rethinking Attention with Performerscited by 1,594
  31. 2019Generating Long Sequences with Sparse Transformerscited by 1,538
  32. 2020Linformer: Self-Attention with Linear Complexitycited by 1,469
  33. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,404
  34. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,365
  35. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,323
  36. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,272
  37. 2023A Survey on Evaluation of Large Language Modelscited by 1,236
  38. 2022LaMDA: Language Models for Dialog Applicationscited by 1,219

50 most cited of 263 All 263 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 1; 2024: 78; 2025: 64; 2026: 42.

185 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 625
  2. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 279
  3. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 235
  4. 2024A Survey on Efficient Inference for Large Language Modelscited by 202
  5. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  6. 2024Large Language Model-Brained GUI Agents: A Surveycited by 161
  7. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 156
  8. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 131
  9. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 126
  10. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 126
  11. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 125
  12. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
38 more
  1. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  2. 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Drivingcited by 85
  3. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 82
  4. 2025Universal Model Routing for Efficient LLM Inferencecited by 82
  5. 2024Frame-Voyager: Learning to Query Frames for Video Large Language Modelscited by 81
  6. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 79
  7. 2024Evaluating Quantized Large Language Modelscited by 76
  8. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 74
  9. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 72
  10. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 67
  11. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learningcited by 55
  12. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 52
  13. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  14. 2024Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learningcited by 47
  15. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 46
  16. 2024Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wildcited by 46
  17. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  18. 2025Agentic AI: A Comprehensive Survey of Architectures, Applications, and Future Directionscited by 46
  19. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 44
  20. 2024Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengthscited by 44
  21. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 44
  22. 2024Combining Fine-Tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justificationscited by 41
  23. 2024TelecomGPT: A Framework to Build Telecom-Specfic Large Language Modelscited by 40
  24. 2025MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMscited by 40
  25. 2024Autoregressive Models in Vision: A Surveycited by 39
  26. 2024A Survey of Deep Learning and Foundation Models for Time Series Forecastingcited by 32
  27. 2024Tiny Titans: Can Smaller Large Language Models Punch Above Their Weight in the Real World for Meeting Summarization?cited by 32
  28. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Modelscited by 31
  29. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  30. 2024MEIT: Multimodal Electrocardiogram Instruction Tuning on Large Language Models for Report Generationcited by 29
  31. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  32. 2024Language Models Need Inductive Biases to Count Inductivelycited by 25
  33. 2024A deeper look at depth pruning of LLMscited by 23
  34. 2025Does Tone Change the Answer? Evaluating Prompt Politeness Effects on Modern LLMs: GPT, Gemini, and LLaMAcited by 23
  35. 2024Self-MoE: Towards Compositional Large Language Models with Self-Specialized Expertscited by 22
  36. 2024Towards Resilient and Efficient LLMs: A Comparative Study of Efficiency, Performance, and Adversarial Robustnesscited by 22
  37. 2024A Survey on the Honesty of Large Language Modelscited by 22
  38. 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Modelscited by 21

50 most cited of 185 All 185 (JSON)