# A Comprehensive Study on Quantization Techniques for Large Language Models

arXiv: https://arxiv.org/abs/2411.02530 · HTML: https://mlsanity.com/paper/2411.02530/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 30 Oct 2024.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023)
- [LLM-QAT: Data-Free Quantization Aware Training for Large Language Models](https://mlsanity.com/paper/2305.17888/) (2023)
- [Compressing Deep Convolutional Networks using Vector Quantization](https://mlsanity.com/paper/1412.6115/) (2014)
- [Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation](https://mlsanity.com/paper/2004.09602/) (2020)
- [EfficientQAT: Efficient Quantization-Aware Training for Large Language Models](https://mlsanity.com/paper/2407.11062/) (2024)
- [BERT: A Review of Applications in Natural Language Processing and Understanding](https://mlsanity.com/paper/2103.11943/) (2021)
- [On the State of the Art of Evaluation in Neural Language Models](https://mlsanity.com/paper/1707.05589/) (2017)
- [Degree-Quant: Quantization-Aware Training for Graph Neural Networks](https://mlsanity.com/paper/2008.05000/) (2020)
- [APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models](https://mlsanity.com/paper/2402.14866/) (2024)
- [MMST-ViT: Climate Change-aware Crop Yield Prediction via Multi-Modal Spatial-Temporal Vision Transformer](https://mlsanity.com/paper/2309.09067/) (2023)
- [Extreme Compression for Pre-trained Transformers Made Simple and Efficient](https://mlsanity.com/paper/2206.01859/) (2022)
- [An Open and Large-Scale Dataset for Multi-Modal Climate Change-aware Crop Yield Predictions](https://mlsanity.com/paper/2406.06081/) (2024)
- [Unveiling Decentralization: A Comprehensive Review of Technologies, Comparison, Challenges in Bitcoin, Ethereum, and Solana Blockchain](https://mlsanity.com/paper/2404.04841/) (2024)
- [Decentralized Adaptive Aerospace Transportation of Unknown Loads Using A Team of Robots](https://mlsanity.com/paper/2407.08084/) (2024)
- [Extracting Attentive Social Temporal Excitation for Sequential Recommendation](https://mlsanity.com/paper/2109.13539/) (2021)
- [BundledSLAM: An Accurate Visual SLAM System Using Multiple Cameras](https://mlsanity.com/paper/2403.19886/) (2024)
- [Advancing Blockchain Scalability: An Introduction to Layer 1 and Layer 2 Solutions](https://mlsanity.com/paper/2406.13855/) (2024)
- [Towards Robust Vision Transformer via Masked Adaptive Ensemble](https://mlsanity.com/paper/2407.15385/) (2024)
- [Advancements in Translation Accuracy for Stereo Visual-Inertial Initialization](https://mlsanity.com/paper/2405.15082/) (2024)

## Built on by (most cited first, 30 of 36)

- [Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization](https://mlsanity.com/paper/2504.09629/) (2025)
- [A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directions](https://mlsanity.com/paper/2412.19823/) (2024)
- [Scaling Laws for Floating Point Quantization Training](https://mlsanity.com/paper/2501.02423/) (2025)
- [GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers](https://mlsanity.com/paper/2506.11784/) (2025)
- [On Jailbreaking Quantized Language Models Through Fault Injection Attacks](https://mlsanity.com/paper/2507.03236/) (2025)
- [Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs](https://mlsanity.com/paper/2604.04722/) (2026)
- [On Accelerating Edge AI: Optimizing Resource-Constrained Environments](https://mlsanity.com/paper/2501.15014/) (2025)
- [Mixed-Precision Quantization for Language Models: Techniques and Prospects](https://mlsanity.com/paper/2510.16805/) (2025)
- [The Devil in the Details: Emergent Misalignment, Format and Coherence in Open-Weights LLMs](https://mlsanity.com/paper/2511.20104/) (2025)
- [Enhancing Ultra-Low-Bit Quantization of Large Language Models Through Saliency-Aware Partial Retraining](https://mlsanity.com/paper/2504.13932/) (2025)
- [ConTextual: Improving Clinical Text Summarization in LLMs with Context-preserving Token Filtering and Knowledge Graphs](https://mlsanity.com/paper/2504.16394/) (2025)
- [MM-Nav: Multi-View VLA Model for Robust Visual Navigation via Multi-Expert Learning](https://mlsanity.com/paper/2510.03142/) (2025)
- [ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generation](https://mlsanity.com/paper/2601.09703/) (2026)
- [Stable FP4 Training via Transposition-Invariant Block Quantization](https://mlsanity.com/paper/2607.24953/) (2026)
- [Intelligent Orchestration of Distributed Large Foundation Model Inference at the Edge](https://mlsanity.com/paper/2504.03668/) (2025)
- [Fine-Tuning Large Language Models and Evaluating Retrieval Methods for Improved Question Answering on Building Codes](https://mlsanity.com/paper/2505.04666/) (2025)
- [RCPU: Rotation-Constrained Error Compensation for Structured Pruning of Large Language Models](https://mlsanity.com/paper/2510.07782/) (2025)
- [Exploiting heterogeneous delays for efficient computation in low-bit neural networks](https://mlsanity.com/paper/2510.27434/) (2025)
- [Joint Partitioning and Placement of Foundation Models for Real-Time Edge AI](https://mlsanity.com/paper/2512.01039/) (2025)
- [From FLOPs to Footprints: The Resource Cost of Artificial Intelligence](https://mlsanity.com/paper/2512.04142/) (2025)
- [OSAQ: Outlier Self-Absorption for Accurate Low-bit LLM Quantization](https://mlsanity.com/paper/2605.04738/) (2026)
- [Resource-Efficient Language Models: Quantization for Fast and Accessible Inference](https://mlsanity.com/paper/2505.08620/) (2025)
- [Pushing the Envelope of LLM Inference with Ultra-Low-Bit Quantized Models](https://mlsanity.com/paper/2508.06753/) (2025)
- [CLARIFY: A Specialist-Generalist Framework for Accurate and Lightweight Dermatological Visual Question Answering](https://mlsanity.com/paper/2508.18430/) (2025)
- [Bridging the Digital Divide: Small Language Models as a Pathway for Physics and Photonics Education in Underdeveloped Regions](https://mlsanity.com/paper/2506.12403/) (2025)
- [READER: Retrieval-Assisted Drafter for Efficient LLM Inference](https://mlsanity.com/paper/2508.09072/) (2025)
- [A Vision Toward Energy-Efficient Domain-Specific Artificial Intelligence Models and Agents](https://mlsanity.com/paper/2510.22052/) (2025)
- [MLPerf Automotive](https://mlsanity.com/paper/2510.27065/) (2025)
- [Opt4GPTQ: Co-Optimizing Memory and Computation for 4-bit GPTQ Quantized LLM Inference on Heterogeneous Platforms](https://mlsanity.com/paper/2511.19438/) (2025)
- [A Systematic Study of Compression Ordering for Large Language Models](https://mlsanity.com/paper/2511.19495/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) have been extensively researched and used in both academia and industry since the rise in popularity of the Transformer model, which demonstrates excellent performance in AI. However, the computational demands of LLMs are immense, and the energy resources required to run them are often limited. For instance, popular models like GPT-3, with 175 billion parameters and a storage requirement of 350 GB, present significant challenges for deployment on resource-constrained IoT devices and embedded systems. These systems often lack the computational capacity to handle such large models. Quantization, a technique that reduces the precision of model values to a smaller set of discrete values, offers a promising solution by reducing the size of LLMs and accelerating inference. In this research, we provide a comprehensive analysis of quantization techniques within the machine learning field, with a particular focus on their application to LLMs. We begin by exploring the mathematical theory of quantization, followed by a review of common quantization methods and how they are implemented. Furthermore, we examine several prominent quantization methods applied to LLMs, detailing their algorithms and performance outcomes.