mlsanityData to 14 Oct 2026

Why Do Some Inputs Break Low-Bit LLM Quantization?

arXiv 2506.12044 ↗First posted 24 May 2025cs.LG cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Low-bit weight-only quantization significantly reduces the memory footprint of large language models (LLMs), but disproportionately affects certain examples. We analyze diverse 3-4 bit methods on LLMs ranging from 7B-70B in size and find that the quantization errors of 50 pairs of methods are strongly correlated (avg. 0.82) on FineWeb examples. Moreover, the residual stream magnitudes of full-precision models are indicative of future quantization errors. We further establish a hypothesis that relates the residual stream magnitudes to error amplification and accumulation over layers. Using LLM localization techniques, early exiting, and activation patching, we show that examples with large errors rely on precise residual activations in the late layers, and that the outputs of MLP gates play a crucial role in maintaining the perplexity. Our work reveals why certain examples result in large quantization errors and which model components are most critical for performance preservation.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 36 of its 49 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

36 of 49 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  8. 2020GLU Variants Improve Transformercited by 1,804
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  10. 2019Root Mean Square Layer Normalizationcited by 1,480
  11. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  12. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
24 more
  1. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  2. 2020On Layer Normalization in the Transformer Architecturecited by 922
  3. 2016Language Modeling with Gated Convolutional Networkscited by 702
  4. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  5. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  6. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  7. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  8. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  9. 2023Llemma: An Open Language Model For Mathematicscited by 388
  10. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  11. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  12. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 155
  13. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
  14. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 149
  15. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 112
  16. 2024Scaling Laws for Precisioncited by 95
  17. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 76
  18. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 61
  19. 2021The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translationcited by 47
  20. 2024How Does Quantization Affect Multilingual LLMs?cited by 43
  21. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkitcited by 33
  22. 2024Fast Matrix Multiplications for Lookup Table-Quantized LLMscited by 29
  23. 2022Intriguing Properties of Compression on Multilingual Modelscited by 15
  24. 2025Fast and Low-Cost Genomic Foundation Models via Outlier Removalcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 8.

9 papers.

  1. 2026Pre-Registering the Detectable Effect: A Paired-MDE Budget for 4-bit Quantization Benchmarks, with a Pilot Auditcited by 8
  2. 2026From Signal Degradation to Computation Collapse: Uncovering the Two Failure Modes of LLM Quantizationcited by 5
  3. 2025LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Modelscited by 3
  4. 2026Beyond Activation Alignment:The Alignment-Diversity Tradeoff in Task-Aware LLM Quantizationcited by 1
  5. 2026Which Decisions Low-Bit Quantization Breaks, and How to Predict Themcited by 1
  6. 2026The Asymmetric Harms of LLM Compressioncited by 1
  7. 2026Why Does Post-Training Quantization Work?cited by 1
  8. 2026When Compression Scores Cannot Decide: Information Boundaries for Group-Robust LLM Pruningcited by 0
  9. 2026Cosine Similarity Is Not Evidence: Measuring the Noise Floor of Interpretability Transfer Under Quantizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Models
  2. 2024What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbation
  3. 2024Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokens
  4. 2025Turning LLM Activations Quantization-Friendly
  5. 2024Understanding the Difficulty of Low-Precision Post-Training Quantization for LLMs
  6. 2025Does quantization affect models' performance on long-context tasks?
  7. 2024Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levels
  8. 2026Quantization Degradation in Large Language Models: A Signal-Noise Perspective