# Benford's Law as a Distributional Prior for Post-Training Quantization of Large Language Models

arXiv: https://arxiv.org/abs/2602.00165 · HTML: https://mlsanity.com/paper/2602.00165/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 45 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [Microscaling Data Formats for Deep Learning](https://mlsanity.com/paper/2310.10537/) (2023)
- [TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate](https://mlsanity.com/paper/2504.19874/) (2025)
- [Convolutional Neural Networks using Logarithmic Data Representation](https://mlsanity.com/paper/1603.01025/) (2016)
- [Post-Training Piecewise Linear Quantization for Deep Neural Networks](https://mlsanity.com/paper/2002.00104/) (2020)
- [Pretraining Large Language Models with NVFP4](https://mlsanity.com/paper/2509.25149/) (2025)
- [A Comprehensive Evaluation of Quantization Strategies for Large Language Models](https://mlsanity.com/paper/2402.16775/) (2024)
- [Norm Tweaking: High-performance Low-bit Quantization of Large Language Models](https://mlsanity.com/paper/2309.02784/) (2023)
- [AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizer](https://mlsanity.com/paper/2407.12951/) (2024)
- [Power-of-Two Quantization for Low Bitwidth and Hardware Compliant Neural Networks](https://mlsanity.com/paper/2203.05025/) (2022)
- [AFPQ: Asymmetric Floating Point Quantization for LLMs](https://mlsanity.com/paper/2311.01792/) (2023)
- [SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights](https://mlsanity.com/paper/2509.22944/) (2025)
- [GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models](https://mlsanity.com/paper/2501.12956/) (2025)

## Abstract (arXiv)

Post-training quantization (PTQ) is a practical way to reduce the memory footprint of large language models, but low-bit quantization is sensitive to mismatches between the quantization codebook and the empirical weight/activation distributions. We revisit Benford-like leading-digit statistics as a lightweight diagnostic of scale-broad behavior in transformer tensors. Across several model families, we observe a consistent functional dichotomy: transformational nn.Linear weights tend to be Benford-like, whereas LayerNorm parameters systematically deviate. Motivated by this observation, we propose BenQ, a data-free PTQ codebook that uses a simple log-spaced grid as a proxy for scale-broad distributions and applies it selectively to transformational layers while keeping stability-critical parameters in higher precision. In 4-bit group-wise PTQ, BenQ consistently improves over uniform RTN and trades wins with NF4 across architectures and tasks, while remaining substantially simpler than optimization-based methods. We additionally report dynamic activation quantization as an exploratory stress test: the results show that log-spaced grids can reduce RTN failures in some families, but also reveal that outlier handling remains essential for reliable low-bit activation PTQ. Code is available at https://github.com/ufopcsilab/benford-quant.