mlsanityData to 14 Oct 2026

The Efficiency Spectrum of Large Language Models: An Algorithmic Survey

arXiv 2312.00678 ↗First posted 1 Dec 2023cs.CLcited by 26

Not on the map. Why?

Abstract (arXiv)

The rapid growth of Large Language Models (LLMs) has been a driving force in transforming various domains, reshaping the artificial general intelligence landscape. However, the increasing computational and memory demands of these models present substantial challenges, hindering both academic research and practical applications. To address these issues, a wide array of methods, including both algorithmic and hardware solutions, have been developed to enhance the efficiency of LLMs. This survey delivers a comprehensive review of algorithmic advancements aimed at improving LLM efficiency. Unlike other surveys that typically focus on specific areas such as training or model compression, this paper examines the multi-faceted dimensions of efficiency essential for the end-to-end algorithmic development of LLMs. Specifically, it covers various topics related to efficiency, including scaling laws, data utilization, architectural innovations, training and tuning strategies, and inference techniques. This paper aims to serve as a valuable resource for researchers and practitioners, laying the groundwork for future innovations in this critical research area. Our repository of relevant references is maintained at url{https://github.com/tding1/Efficient-LLM-Survey}.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 218 of its 319 references that are in the map. First posted on arXiv 1 Dec 2023.

Builds on

218 of 319 references.

References · 218

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  10. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  11. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  12. 2020Scaling Laws for Neural Language Modelscited by 6,586
38 more
  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2017Deep reinforcement learning from human preferencescited by 3,912
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  13. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  14. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  15. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  16. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  17. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  18. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  19. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  20. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  21. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  22. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  23. 2018Deep contextualized word representationscited by 2,290
  24. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  25. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  26. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  27. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  28. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  29. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  30. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  31. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  32. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  33. 2020Reformer: The Efficient Transformercited by 1,668
  34. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  35. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  36. 2020Knowledge Distillation: A Surveycited by 1,603
  37. 2020Rethinking Attention with Performerscited by 1,601
  38. 2019Generating Long Sequences with Sparse Transformerscited by 1,546

50 most cited of 218 All 218 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 12; 2025: 6; 2026: 5.

26 papers.

Sort:

By area:

  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  2. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  3. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  4. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  5. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  6. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  7. 2024PaCE: Parsimonious Concept Engineering for Large Language Modelscited by 19
  8. 2024Data Efficient Evaluation of Large Language Models and Text-to-Image Models via Adaptive Samplingcited by 12
  9. 2024Are Small Language Models Ready to Compete with Large Language Models for Practical Applications?cited by 9
  10. 2023OTOv3: Automatic Architecture-Agnostic Neural Network Training and Compression from Structured Pruning to Erasing Operatorscited by 6
  11. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniquescited by 5
  12. 2025TokenSqueeze: Performance-Preserving Compression for Reasoning LLMscited by 5
14 more
  1. 2024HESSO: Towards Automatic Efficient and User Friendly Any Neural Network Training and Pruningcited by 4
  2. 2024Quasi-random Multi-Sample Inference for Large Language Modelscited by 4
  3. 2026From AGI to ASIcited by 4
  4. 2023Robust Neural Pruning with Gradient Sampling Optimization for Residual Neural Networkscited by 3
  5. 2025WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inferencecited by 3
  6. 2025Exploiting Unstructured Sparsity in Fully Homomorphic Encrypted DNNscited by 2
  7. 2026StableQAT: Stable Quantization-Aware Training at Ultra-Low Bitwidthscited by 2
  8. 2024All Language Models Large and Smallcited by 1
  9. 2025MeTA-LoRA: Data-Efficient Multi-Task Fine-Tuning for Large Language Modelscited by 1
  10. 2024CIKMar: A Dual-Encoder Approach to Prompt-Based Reranking in Educational Dialogue Systemscited by 0
  11. 2024KLCBL: An Improved Police Incident Classification Modelcited by 0
  12. 2026How Is Uncertainty Propagated in Knowledge Distillation?cited by 0
  13. 2026Grounded Scaling: Why Agentic AI Needs Deterministic Environmentscited by 0
  14. 2026Performance Foundations of Parallel & Distributed Reasoning Language Modelscited by 0