# Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokens

arXiv: https://arxiv.org/abs/2411.17691 · HTML: https://mlsanity.com/paper/2411.17691/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 31 references that are in the map.
- First posted on arXiv 26 Nov 2024.

## Builds on (most cited first, 29 of 29)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://mlsanity.com/paper/2402.02750/) (2024)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [Q8BERT: Quantized 8Bit BERT](https://mlsanity.com/paper/1910.06188/) (2019)
- [Extreme Compression of Large Language Models via Additive Quantization](https://mlsanity.com/paper/2401.06118/) (2024)
- [BitNet: Scaling 1-bit Transformers for Large Language Models](https://mlsanity.com/paper/2310.11453/) (2023)
- [BinaryBERT: Pushing the Limit of BERT Quantization](https://mlsanity.com/paper/2012.15701/) (2020)
- [TernaryBERT: Distillation-aware Ultra-low Bit BERT](https://mlsanity.com/paper/2009.12812/) (2020)
- [BiLLM: Pushing the Limit of Post-Training Quantization for LLMs](https://mlsanity.com/paper/2402.04291/) (2024)
- [Scaling Laws for Precision](https://mlsanity.com/paper/2411.04330/) (2024)
- [QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models](https://mlsanity.com/paper/2310.08041/) (2023)
- [LQER: Low-Rank Quantization Error Reconstruction for LLMs](https://mlsanity.com/paper/2402.02446/) (2024)
- [Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge](https://mlsanity.com/paper/2312.05693/) (2023)
- [How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMs](https://mlsanity.com/paper/2410.13857/) (2024)
- [Towards Accurate Post-Training Quantization of Vision Transformers via Error Reduction](https://mlsanity.com/paper/2407.06794/) (2024)
- [Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale](https://mlsanity.com/paper/2407.12327/) (2024)

## Built on by (most cited first, 19 of 19)

- [Scaling Law for Quantization-Aware Training](https://mlsanity.com/paper/2505.14302/) (2025)
- [Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs](https://mlsanity.com/paper/2502.12982/) (2025)
- [Training Dynamics Impact Post-Training Quantization Robustness](https://mlsanity.com/paper/2510.06213/) (2025)
- [Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysis](https://mlsanity.com/paper/2502.13178/) (2025)
- [Hyperloop Transformers](https://mlsanity.com/paper/2604.21254/) (2026)
- [QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2502.12346/) (2025)
- [Scaling Laws for Floating Point Quantization Training](https://mlsanity.com/paper/2501.02423/) (2025)
- [Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory](https://mlsanity.com/paper/2510.12077/) (2025)
- [Characterization and Mitigation of Training Instabilities in Microscaling Formats](https://mlsanity.com/paper/2506.20752/) (2025)
- [Scaling Laws for Precision in High-Dimensional Linear Regression](https://mlsanity.com/paper/2602.19241/) (2026)
- [Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Models](https://mlsanity.com/paper/2508.06974/) (2025)
- [Compressed code: the hidden effects of quantization and distillation on programming tokens](https://mlsanity.com/paper/2601.02563/) (2026)
- [Which Decisions Low-Bit Quantization Breaks, and How to Predict Them](https://mlsanity.com/paper/2608.06564/) (2026)
- [Capability-Stratified Degradation in Ternary Language Models](https://mlsanity.com/paper/2608.28809/) (2026)
- [LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillation](https://mlsanity.com/paper/2604.19167/) (2026)
- [FTerViT: Fully Ternary Vision Transformer](https://mlsanity.com/paper/2605.21171/) (2026)
- [LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws](https://mlsanity.com/paper/2605.23901/) (2026)
- [On the Expressive Power of Weight Quantization in Large Language Models](https://mlsanity.com/paper/2606.22249/) (2026)
- [Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference](https://mlsanity.com/paper/2609.26621/) (2026)

## Abstract (arXiv)

We reveal that low-bit quantization favors undertrained large language models (LLMs) by observing that models with larger sizes or fewer training tokens experience less quantization-induced degradation (QiD) when applying low-bit quantization, whereas smaller models with extensive training tokens suffer significant QiD. To gain deeper insights into this trend, we study over 1500 quantized LLM checkpoints of various sizes and at different training levels (undertrained or fully trained) in a controlled setting, deriving scaling laws for understanding the relationship between QiD and factors such as the number of training tokens, model size and bit width. With the derived scaling laws, we propose a novel perspective that we can use QiD to measure an LLM's training levels and determine the number of training tokens required for fully training LLMs of various sizes. Moreover, we use the scaling laws to predict the quantization performance of different-sized LLMs trained with 100 trillion tokens. Our projection shows that the low-bit quantization performance of future models, which are expected to be trained with over 100 trillion tokens, may NOT be desirable. This poses a potential challenge for low-bit quantization in the future and highlights the need for awareness of a model's training level when evaluating low-bit quantization research. To facilitate future research on this problem, we release all the 1500+ quantized checkpoints used in this work at https://huggingface.co/Xu-Ouyang.