mlsanityData to 7 Oct 2026

The case for 4-bit precision: k-bit Inference Scaling Laws

arXiv 2212.09720 ↗First posted 19 Dec 2022cs.LG cs.NEcited by 247

Not on the map. Why?

Abstract (arXiv)

Quantization methods reduce the number of bits required to represent each parameter in a model, trading accuracy for smaller memory footprints and inference latencies. However, the final model size depends on both the number of parameters of the original model and the rate of compression. For example, a 30B 8-bit model and a 60B 4-bit model have the same number of bits but may have very different zero-shot accuracies. In this work, we study this trade-off by developing inference scaling laws of zero-shot performance in Large Language Models (LLMs) to determine the bit-precision and model size that maximizes zero-shot performance. We run more than 35,000 experiments with 16-bit inputs and k-bit parameters to examine which zero-shot quantization methods improve scaling for 3 to 8-bit precision at scales of 19M to 176B parameters across the LLM families BLOOM, OPT, NeoX/Pythia, and GPT-2. We find that it is challenging to improve the bit-level scaling trade-off, with the only improvements being the use of a small block size -- splitting the parameters into small independently quantized blocks -- and the quantization data type being used (e.g., Int vs Float). Overall, our findings show that {4-bit} precision is almost universally optimal for total model bits and zero-shot accuracy.

Key facts. Cited by 247 papers in the map (data through 7 Oct 2026). Builds on 43 of its 55 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

43 of 55 references.

References · 43

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  9. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  11. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  12. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
31 more
  1. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  2. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  3. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  4. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  5. 2022Efficiently Scaling Transformer Inferencecited by 529
  6. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  7. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  8. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  9. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  10. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 328
  11. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 272
  12. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  13. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  14. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 189
  15. 2022FP8 Formats for Deep Learningcited by 187
  16. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 183
  17. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  18. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  19. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  20. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  21. 2019Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networkscited by 87
  22. 2021Sparse is Enough in Scaling Transformerscited by 86
  23. 2017Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMMcited by 60
  24. 20158-Bit Approximations for Parallelism in Deep Learningcited by 56
  25. 2016Loss-aware Binarization of Deep Networkscited by 56
  26. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  27. 2016Towards the Limit of Network Quantizationcited by 49
  28. 2020Positional Artefacts Propagate Through Masked Language Model Embeddingscited by 34
  29. 2019Memory-Driven Mixed Low Precision Quantization For Enabling Deep Network Inference On Microcontrollerscited by 31
  30. 2022F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantizationcited by 26
  31. 2019Dissecting the NVidia Turing T4 GPU via Microbenchmarkingcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 62; 2024: 95; 2025: 59; 2026: 31.

247 papers.

Sort:

By area:

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  3. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  5. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  7. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  8. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  9. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  10. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  11. 2023A Survey on Model Compression for Large Language Modelscited by 263
  12. 2023Pretraining Language Models with Human Preferencescited by 245
38 more
  1. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  2. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  3. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  4. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  5. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
  6. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  7. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
  8. 2023ChatEDA: A Large Language Model Powered Autonomous Agent for EDAcited by 118
  9. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 108
  10. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  11. 2024Scaling Laws for Precisioncited by 92
  12. 2023Fast Inference of Mixture-of-Experts Language Models with Offloadingcited by 88
  13. 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 86
  14. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 80
  15. 2023MatFormer: Nested Transformer for Elastic Inferencecited by 79
  16. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 78
  17. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 77
  18. 2024Can large language models explore in-context?cited by 75
  19. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  20. 2023Inference with Reference: Lossless Acceleration of Large Language Modelscited by 63
  21. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 62
  22. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  23. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  24. 2023Stateful Large Language Model Serving with Pensievecited by 59
  25. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 59
  26. 2023Memory Efficient Optimizers with 4-bit Statescited by 54
  27. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 54
  28. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 53
  29. 2023On the Impact of Calibration Data in Post-training Quantization and Pruningcited by 51
  30. 2024The Super Weight in Large Language Modelscited by 51
  31. 2023Intriguing Properties of Quantization at Scalecited by 48
  32. 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
  33. 2024MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Modelscited by 48
  34. 2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 47
  35. 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 47
  36. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devicescited by 45
  37. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 45
  38. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formatscited by 44

50 most cited of 247 All 247 (JSON)