# A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms

arXiv: https://arxiv.org/abs/2409.16694 · HTML: https://mlsanity.com/paper/2409.16694/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 69 of its 114 references that are in the map.
- First posted on arXiv 25 Sep 2024.

## Builds on (most cited first, 30 of 69)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://mlsanity.com/paper/2404.00456/) (2024)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models](https://mlsanity.com/paper/2308.13137/) (2023)
- [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023)
- [8-bit Optimizers via Block-wise Quantization](https://mlsanity.com/paper/2110.02861/) (2021)
- [QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks](https://mlsanity.com/paper/2402.04396/) (2024)
- [SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression](https://mlsanity.com/paper/2306.03078/) (2023)
- [Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT](https://mlsanity.com/paper/1909.05840/) (2019)
- [Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning](https://mlsanity.com/paper/2208.11580/) (2022)
- [SqueezeLLM: Dense-and-Sparse Quantization](https://mlsanity.com/paper/2306.07629/) (2023)
- [Atom: Low-bit Quantization for Efficient and Accurate LLM Serving](https://mlsanity.com/paper/2310.19102/) (2023)
- [Extreme Compression of Large Language Models via Additive Quantization](https://mlsanity.com/paper/2401.06118/) (2024)
- [LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models](https://mlsanity.com/paper/2310.08659/) (2023)
- [BitNet: Scaling 1-bit Transformers for Large Language Models](https://mlsanity.com/paper/2310.11453/) (2023)
- [Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation](https://mlsanity.com/paper/2004.09602/) (2020)
- [Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models](https://mlsanity.com/paper/2209.13325/) (2022)
- [QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization](https://mlsanity.com/paper/2203.05740/) (2022)
- [PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantization](https://mlsanity.com/paper/2111.12293/) (2021)
- [BinaryBERT: Pushing the Limit of BERT Quantization](https://mlsanity.com/paper/2012.15701/) (2020)

## Built on by (most cited first, 30 of 31)

- [Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization](https://mlsanity.com/paper/2504.09629/) (2025)
- [An Empirical Study of Qwen3 Quantization](https://mlsanity.com/paper/2505.02214/) (2025)
- [Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysis](https://mlsanity.com/paper/2502.13178/) (2025)
- [Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attention](https://mlsanity.com/paper/2602.04789/) (2026)
- [BiDM: Pushing the Limit of Quantization for Diffusion Models](https://mlsanity.com/paper/2412.05926/) (2024)
- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [LittleBit: Ultra Low-Bit Quantization via Latent Factorization](https://mlsanity.com/paper/2506.13771/) (2025)
- [First-Order Error Matters: Accurate Compensation for Quantized Large Language Models](https://mlsanity.com/paper/2507.11017/) (2025)
- [SGMD: Score Gradient Matching Distillation for Few-Step Video Diffusion Distillation](https://mlsanity.com/paper/2605.30116/) (2026)
- [Mixed-Precision Quantization for Language Models: Techniques and Prospects](https://mlsanity.com/paper/2510.16805/) (2025)
- [DynaMo: Runtime Switchable Quantization for MoE with Cross-Dataset Adaptation](https://mlsanity.com/paper/2503.21135/) (2025)
- [Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression](https://mlsanity.com/paper/2510.20984/) (2025)
- [HBLLM: Wavelet-Enhanced High-Fidelity 1-Bit Quantization for LLMs](https://mlsanity.com/paper/2512.00862/) (2025)
- [The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inference](https://mlsanity.com/paper/2603.19664/) (2026)
- [Membership Inference Risks in Quantized Models: A Theoretical and Empirical Study](https://mlsanity.com/paper/2502.06567/) (2025)
- [OneComp: One-Line Revolution for Generative AI Model Compression](https://mlsanity.com/paper/2603.28845/) (2026)
- [Stable FP4 Training via Transposition-Invariant Block Quantization](https://mlsanity.com/paper/2607.24953/) (2026)
- [FedHQ: Hybrid Runtime Quantization for Federated Learning](https://mlsanity.com/paper/2505.11982/) (2025)
- [BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Models](https://mlsanity.com/paper/2602.04163/) (2026)
- [Resource-Efficient Language Models: Quantization for Fast and Accessible Inference](https://mlsanity.com/paper/2505.08620/) (2025)
- [Reasoning Language Model Inference Serving Unveiled: An Empirical Study](https://mlsanity.com/paper/2510.18672/) (2025)
- [Rudder: Steering Prefetching in Distributed GNN Training using LLM Agents](https://mlsanity.com/paper/2602.23556/) (2026)
- [ReQuant: Fixed-Grid Discrete Refinement for Post-Training Quantization](https://mlsanity.com/paper/2608.07019/) (2026)
- [Sliced-Wasserstein Distribution Alignment Loss Improves the Ultra-Low-Bit Quantization of Large Language Models](https://mlsanity.com/paper/2601.07878/) (2026)
- [Fine-Grained Model Merging via Modular Expert Recombination](https://mlsanity.com/paper/2602.06552/) (2026)
- [Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression](https://mlsanity.com/paper/2606.18304/) (2026)
- [An Empirical Study of openPangu Quantization on Ascend NPUs](https://mlsanity.com/paper/2606.21257/) (2026)
- [BitNet Text Embeddings](https://mlsanity.com/paper/2606.25674/) (2026)
- [One Qubit Can Beat One Bit: Quantum Advantage for Post-Training Quantization](https://mlsanity.com/paper/2608.05240/) (2026)
- [Quantization Degradation in Large Language Models: A Signal-Noise Perspective](https://mlsanity.com/paper/2608.08188/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have achieved remarkable advancements in natural language processing, showcasing exceptional performance across various tasks. However, the expensive memory and computational requirements present significant challenges for their practical deployment. Low-bit quantization has emerged as a critical approach to mitigate these challenges by reducing the bit-width of model parameters, activations, and gradients, thus decreasing memory usage and computational demands. This paper presents a comprehensive survey of low-bit quantization methods tailored for LLMs, covering the fundamental principles, system implementations, and algorithmic strategies. An overview of basic concepts and new data formats specific to low-bit LLMs is first introduced, followed by a review of frameworks and systems that facilitate low-bit LLMs across various hardware platforms. Then, we categorize and analyze techniques and toolkits for efficient low-bit training and inference of LLMs. Finally, we conclude with a discussion of future trends and potential advancements of low-bit LLMs. Our systematic overview from basic, system, and algorithm perspectives can offer valuable insights and guidelines for future works to enhance the efficiency and applicability of LLMs through low-bit quantization.