mlsanityData to 7 Oct 2026

Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward

arXiv 2402.01799 ↗First posted 2 Feb 2024cs.LG cs.AI cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

Despite the impressive performance of LLMs, their widespread adoption faces challenges due to substantial computational and memory requirements during inference. Recent advancements in model compression and system-level optimization methods aim to enhance LLM inference. This survey offers an overview of these methods, emphasizing recent developments. Through experiments on LLaMA(/2)-7B, we evaluate various compression techniques, providing practical insights for efficient LLM deployment in a unified setting. The empirical analysis on LLaMA(/2)-7B highlights the effectiveness of these methods. Drawing from survey insights, we identify current limitations and discuss potential future directions to improve LLM inference efficiency. We release the codebase to reproduce the results presented in this paper at https://github.com/nyunAI/Faster-LLM-Survey

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 49 of its 60 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

49 of 60 references.

References · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  12. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
37 more
  1. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  2. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  3. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  4. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  5. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  6. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  7. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  8. 2019Learned Step Size Quantizationcited by 604
  9. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  10. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  11. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  12. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  13. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  14. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  15. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  16. 2023A Survey on Model Compression for Large Language Modelscited by 263
  17. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  18. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
  19. 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 185
  20. 2021Block Pruning For Faster Transformerscited by 176
  21. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  22. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 120
  23. 2023Fluctuation-based Adaptive Structured Pruning for Large Language Modelscited by 110
  24. 2023SCOTT: Self-Consistent Chain-of-Thought Distillationcited by 103
  25. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  26. 2015Convolutional neural networks with low-rank regularizationcited by 102
  27. 2020Degree-Quant: Quantization-Aware Training for Graph Neural Networkscited by 93
  28. 2022Less is More: Task-aware Layer-wise Distillation for Language Model Compressioncited by 91
  29. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  30. 2022Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Spacecited by 66
  31. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formatscited by 44
  32. 2023LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recoverycited by 37
  33. 2023Lion: Adversarial Distillation of Proprietary Large Language Modelscited by 33
  34. 2023LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compressioncited by 26
  35. 2023One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Modelscited by 16
  36. 2020WrapNet: Neural Net Inference with Ultra-Low-Resolution Arithmeticcited by 12
  37. 2022Pruning Pre-trained Language Models Without Fine-Tuningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 12; 2026: 2.

20 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Models for Code Generationcited by 646
  2. 2024MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Modelscited by 48
  3. 2025Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applicationscited by 26
  4. 2024Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applicationscited by 25
  5. 2025Enhancing Autonomous Driving Systems with On-Board Deployed Large Language Modelscited by 11
  6. 2024Grammar-based Game Description Generation using Large Language Modelscited by 7
  7. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniquescited by 5
  8. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Modelscited by 5
  9. 2025Edge Deployment of Small Language Models, a comprehensive comparison of CPU, GPU and NPU backendscited by 4
  10. 2026PowLU: An Activation Function for Stable Pre-Training of LLMscited by 4
  11. 2025DReSS: Data-driven Regularized Structured Streamlining for Large Language Modelscited by 3
  12. 2025Incomplete Graph Learning: A Comprehensive Surveycited by 3
8 more
  1. 2024From Cool Demos to Production-Ready FMware: Core Challenges and a Technology Roadmapcited by 2
  2. 2025From Raw Corpora to Domain Benchmarks: Automated Evaluation of LLM Domain Expertisecited by 1
  3. 2025FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmeticcited by 1
  4. 2024Enhancing Knowledge Distillation for LLMs with Response-Priming Promptingcited by 0
  5. 2025HALO: Memory-Centric Heterogeneous Accelerator with 2.5D Integration for Low-Batch LLM Inferencecited by 0
  6. 2025SQ-format: A Unified Sparse-Quantized Hardware-friendly Data Format for LLMscited by 0
  7. 2025Toward Agentic Environments: GenAI and the Convergence of AI, Sustainability, and Human-Centric Spacescited by 0
  8. 2026Meeting SLOs, Slashing Hours: Automated Enterprise LLM Optimization with OptiKITcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LLM Inference Unveiled: Survey and Roofline Model Insights
  2. 2025Huff-LLM: End-to-End Lossless Compression for Efficient LLM Inference
  3. 2026Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference
  4. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques
  5. 2023A Speed Odyssey for Deployable Quantization of LLMs
  6. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline
  7. 2025Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design
  8. 2023Efficient LLM Inference on CPUs