mlsanityData to 14 Oct 2026

Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerations

arXiv 2408.03130 ↗First posted 6 Aug 2024cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Large language models are ubiquitous in natural language processing because they can adapt to new tasks without retraining. However, their sheer scale and complexity present unique challenges and opportunities, prompting researchers and practitioners to explore novel model training, optimization, and deployment methods. This literature review focuses on various techniques for reducing resource requirements and compressing large language models, including quantization, pruning, knowledge distillation, and architectural optimizations. The primary objective is to explore each method in-depth and highlight its unique challenges and practical applications. The discussed methods are categorized into a taxonomy that presents an overview of the optimization landscape and helps navigate it to understand the research trajectory better.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 69 of its 83 references that are in the map. First posted on arXiv 6 Aug 2024.

Builds on

69 of 83 references.

References · 69

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  9. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  11. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  12. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
38 more
  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2020Longformer: The Long-Document Transformercited by 3,413
  6. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  7. 2022Emergent Abilities of Large Language Modelscited by 2,512
  8. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  12. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  13. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  14. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  15. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  16. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  17. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  18. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  19. 2023StarCoder: may the source be with you!cited by 799
  20. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  21. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  22. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  23. 2013Do Deep Nets Really Need to be Deep?cited by 474
  24. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  25. 2022Decoupled Knowledge Distillationcited by 388
  26. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  27. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  28. 2022Large Language Models Are Reasoning Teacherscited by 329
  29. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  30. 2022Teaching Small Language Models to Reasoncited by 284
  31. 2023The False Promise of Imitating Proprietary LLMscited by 263
  32. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  33. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 251
  34. 2014cuDNN: Efficient Primitives for Deep Learningcited by 245
  35. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 188
  36. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
  37. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 158
  38. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 139

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decodingcited by 9
  2. 2025Streaming Looking Ahead with Token-level Self-rewardcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Efficient Inference for Large Language Models
  2. 2024Model Compression and Efficient Inference for Large Language Models: A Survey
  3. 2023A Survey on Model Compression for Large Language Models
  4. 2024Efficiency optimization of large-scale language models based on deep learning in natural language processing tasks
  5. 2025Resource-Efficient Language Models: Quantization for Fast and Accessible Inference
  6. 2025Systematic Evaluation of Optimization Techniques for Long-Context Language Models
  7. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques
  8. 2025Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models