mlsanityData to 7 Oct 2026

A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms

arXiv 2409.16694 ↗First posted 25 Sep 2024cs.AI cs.CL cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have achieved remarkable advancements in natural language processing, showcasing exceptional performance across various tasks. However, the expensive memory and computational requirements present significant challenges for their practical deployment. Low-bit quantization has emerged as a critical approach to mitigate these challenges by reducing the bit-width of model parameters, activations, and gradients, thus decreasing memory usage and computational demands. This paper presents a comprehensive survey of low-bit quantization methods tailored for LLMs, covering the fundamental principles, system implementations, and algorithmic strategies. An overview of basic concepts and new data formats specific to low-bit LLMs is first introduced, followed by a review of frameworks and systems that facilitate low-bit LLMs across various hardware platforms. Then, we categorize and analyze techniques and toolkits for efficient low-bit training and inference of LLMs. Finally, we conclude with a discussion of future trends and potential advancements of low-bit LLMs. Our systematic overview from basic, system, and algorithm perspectives can offer valuable insights and guidelines for future works to enhance the efficiency and applicability of LLMs through low-bit quantization.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 69 of its 114 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

69 of 114 references.

References · 69

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  10. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  11. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  12. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
38 more
  1. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  2. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  3. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  4. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  5. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  6. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  7. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  8. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  9. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  10. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  11. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  12. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  13. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 183
  14. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  15. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  16. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 175
  17. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  18. 2021PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantizationcited by 173
  19. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  20. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  21. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  22. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  23. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 137
  24. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 112
  25. 2024Higher Layers Need More LoRA Expertscited by 101
  26. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  27. 2022FP8 Quantization: The Power of the Exponentcited by 87
  28. 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 86
  29. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 83
  30. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 78
  31. 2019Towards Unified INT8 Training for Convolutional Neural Networkcited by 77
  32. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 74
  33. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  34. 2023Memory Efficient Optimizers with 4-bit Statescited by 54
  35. 2023FlexRound: Learnable Rounding based on Element-wise Division for Post-Training Quantizationcited by 46
  36. 2022ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantizationcited by 42
  37. 2024Low-Rank Quantization-Aware Training for LLMscited by 42
  38. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Modelscited by 41

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 14; 2026: 15.

30 papers.

Sort:

By area:

  1. 2025Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantizationcited by 36
  2. 2025An Empirical Study of Qwen3 Quantizationcited by 20
  3. 2025Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysiscited by 12
  4. 2024BiDM: Pushing the Limit of Quantization for Diffusion Modelscited by 11
  5. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  6. 2025LittleBit: Ultra Low-Bit Quantization via Latent Factorizationcited by 11
  7. 2026Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attentioncited by 9
  8. 2025First-Order Error Matters: Accurate Compensation for Quantized Large Language Modelscited by 8
  9. 2026SGMD: Score Gradient Matching Distillation for Few-Step Video Diffusion Distillationcited by 6
  10. 2025DynaMo: Runtime Switchable Quantization for MoE with Cross-Dataset Adaptationcited by 4
  11. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 4
  12. 2025Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compressioncited by 4
18 more
  1. 2025HBLLM: Wavelet-Enhanced High-Fidelity 1-Bit Quantization for LLMscited by 4
  2. 2026The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inferencecited by 4
  3. 2025Membership Inference Risks in Quantized Models: A Theoretical and Empirical Studycited by 3
  4. 2026OneComp: One-Line Revolution for Generative AI Model Compressioncited by 3
  5. 2026Stable FP4 Training via Transposition-Invariant Block Quantizationcited by 3
  6. 2025FedHQ: Hybrid Runtime Quantization for Federated Learningcited by 2
  7. 2025Resource-Efficient Language Models: Quantization for Fast and Accessible Inferencecited by 1
  8. 2025Reasoning Language Model Inference Serving Unveiled: An Empirical Studycited by 1
  9. 2026BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Modelscited by 1
  10. 2026Sliced-Wasserstein Distribution Alignment Loss Improves the Ultra-Low-Bit Quantization of Large Language Modelscited by 0
  11. 2026Fine-Grained Model Merging via Modular Expert Recombinationcited by 0
  12. 2026Rudder: Steering Prefetching in Distributed GNN Training using LLM Agentscited by 0
  13. 2026Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compressioncited by 0
  14. 2026An Empirical Study of openPangu Quantization on Ascend NPUscited by 0
  15. 2026BitNet Text Embeddingscited by 0
  16. 2026One Qubit Can Beat One Bit: Quantum Advantage for Post-Training Quantizationcited by 0
  17. 2026ReQuant: Fixed-Grid Discrete Refinement for Post-Training Quantizationcited by 0
  18. 2026Quantization Degradation in Large Language Models: A Signal-Noise Perspectivecited by 0