mlsanityData to 14 Oct 2026

A Comprehensive Study on Quantization Techniques for Large Language Models

arXiv 2411.02530 ↗First posted 30 Oct 2024cs.LG cs.AI cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have been extensively researched and used in both academia and industry since the rise in popularity of the Transformer model, which demonstrates excellent performance in AI. However, the computational demands of LLMs are immense, and the energy resources required to run them are often limited. For instance, popular models like GPT-3, with 175 billion parameters and a storage requirement of 350 GB, present significant challenges for deployment on resource-constrained IoT devices and embedded systems. These systems often lack the computational capacity to handle such large models. Quantization, a technique that reduces the precision of model values to a smaller set of discrete values, offers a promising solution by reducing the size of LLMs and accelerating inference. In this research, we provide a comprehensive analysis of quantization techniques within the machine learning field, with a particular focus on their application to LLMs. We begin by exploring the mathematical theory of quantization, followed by a review of common quantization methods and how they are implemented. Furthermore, we examine several prominent quantization methods applied to LLMs, detailing their algorithms and performance outcomes.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 26 of its 35 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

26 of 35 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  3. 2023A Survey of Large Language Modelscited by 2,576
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  7. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  8. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  9. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  10. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 195
  11. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 188
  12. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
14 more
  1. 2021BERT: A Review of Applications in Natural Language Processing and Understandingcited by 109
  2. 2017On the State of the Art of Evaluation in Neural Language Modelscited by 104
  3. 2020Degree-Quant: Quantization-Aware Training for Graph Neural Networkscited by 96
  4. 2024APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Modelscited by 34
  5. 2023MMST-ViT: Climate Change-aware Crop Yield Prediction via Multi-Modal Spatial-Temporal Vision Transformercited by 29
  6. 2022Extreme Compression for Pre-trained Transformers Made Simple and Efficientcited by 19
  7. 2024An Open and Large-Scale Dataset for Multi-Modal Climate Change-aware Crop Yield Predictionscited by 11
  8. 2024Unveiling Decentralization: A Comprehensive Review of Technologies, Comparison, Challenges in Bitcoin, Ethereum, and Solana Blockchaincited by 6
  9. 2024Decentralized Adaptive Aerospace Transportation of Unknown Loads Using A Team of Robotscited by 6
  10. 2021Extracting Attentive Social Temporal Excitation for Sequential Recommendationcited by 5
  11. 2024BundledSLAM: An Accurate Visual SLAM System Using Multiple Camerascited by 4
  12. 2024Advancing Blockchain Scalability: An Introduction to Layer 1 and Layer 2 Solutionscited by 4
  13. 2024Towards Robust Vision Transformer via Masked Adaptive Ensemblecited by 4
  14. 2024Advancements in Translation Accuracy for Stereo Visual-Inertial Initializationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 26; 2026: 9.

36 papers.

Sort:

By area:

  1. 2025Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantizationcited by 41
  2. 2024A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directionscited by 25
  3. 2025Scaling Laws for Floating Point Quantization Trainingcited by 8
  4. 2025GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformerscited by 8
  5. 2025On Jailbreaking Quantized Language Models Through Fault Injection Attackscited by 8
  6. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMscited by 7
  7. 2025On Accelerating Edge AI: Optimizing Resource-Constrained Environmentscited by 5
  8. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 5
  9. 2025The Devil in the Details: Emergent Misalignment, Format and Coherence in Open-Weights LLMscited by 5
  10. 2025Enhancing Ultra-Low-Bit Quantization of Large Language Models Through Saliency-Aware Partial Retrainingcited by 3
  11. 2025ConTextual: Improving Clinical Text Summarization in LLMs with Context-preserving Token Filtering and Knowledge Graphscited by 3
  12. 2025MM-Nav: Multi-View VLA Model for Robust Visual Navigation via Multi-Expert Learningcited by 3
24 more
  1. 2026ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generationcited by 3
  2. 2026Stable FP4 Training via Transposition-Invariant Block Quantizationcited by 3
  3. 2025Intelligent Orchestration of Distributed Large Foundation Model Inference at the Edgecited by 2
  4. 2025Fine-Tuning Large Language Models and Evaluating Retrieval Methods for Improved Question Answering on Building Codescited by 2
  5. 2025RCPU: Rotation-Constrained Error Compensation for Structured Pruning of Large Language Modelscited by 2
  6. 2025Exploiting heterogeneous delays for efficient computation in low-bit neural networkscited by 2
  7. 2025Joint Partitioning and Placement of Foundation Models for Real-Time Edge AIcited by 2
  8. 2025From FLOPs to Footprints: The Resource Cost of Artificial Intelligencecited by 2
  9. 2026OSAQ: Outlier Self-Absorption for Accurate Low-bit LLM Quantizationcited by 2
  10. 2025Resource-Efficient Language Models: Quantization for Fast and Accessible Inferencecited by 1
  11. 2025Pushing the Envelope of LLM Inference with Ultra-Low-Bit Quantized Modelscited by 1
  12. 2025CLARIFY: A Specialist-Generalist Framework for Accurate and Lightweight Dermatological Visual Question Answeringcited by 1
  13. 2025Bridging the Digital Divide: Small Language Models as a Pathway for Physics and Photonics Education in Underdeveloped Regionscited by 0
  14. 2025READER: Retrieval-Assisted Drafter for Efficient LLM Inferencecited by 0
  15. 2025A Vision Toward Energy-Efficient Domain-Specific Artificial Intelligence Models and Agentscited by 0
  16. 2025MLPerf Automotivecited by 0
  17. 2025Opt4GPTQ: Co-Optimizing Memory and Computation for 4-bit GPTQ Quantized LLM Inference on Heterogeneous Platformscited by 0
  18. 2025A Systematic Study of Compression Ordering for Large Language Modelscited by 0
  19. 2025Evaluating Small Language Models for Agentic On-Farm Decision Support Systemscited by 0
  20. 2026Sliced-Wasserstein Distribution Alignment Loss Improves the Ultra-Low-Bit Quantization of Large Language Modelscited by 0
  21. 2026Weight Group-wise Post-Training Quantization for Medical Foundation Modelcited by 0
  22. 2026On the Invariance and Generality of Neural Scaling Lawscited by 0
  23. 2026FlipGuard: Defending Large Language Models Against Quantization-Conditioned Backdoor Attackscited by 0
  24. 2026When Personality Meets Quantization: A Layer-wise MBTI Analysis of Quantized LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox
  2. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Models
  3. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit
  4. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms
  5. 2025A Comprehensive Evaluation on Quantization Techniques for Large Language Models
  6. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques
  7. 2025Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective
  8. 2024Continuous Approximations for Improving Quantization Aware Training of LLMs