mlsanityData to 13 Oct 2026

Intriguing Properties of Quantization at Scale

arXiv 2305.19268 ↗First posted 30 May 2023cs.LG cs.AIcited by 49

Not on the map. Why?

Abstract (arXiv)

Emergent properties have been widely adopted as a term to describe behavior not present in smaller models but observed in larger models. Recent work suggests that the trade-off incurred by quantization is also an emergent property, with sharp drops in performance in models over 6B parameters. In this work, we ask "are quantization cliffs in performance solely a factor of scale?" Against a backdrop of increased research focus on why certain emergent properties surface at scale, this work provides a useful counter-example. We posit that it is possible to optimize for a quantization friendly training recipe that suppresses large activation magnitude outliers. Here, we find that outlier dimensions are not an inherent product of scale, but rather sensitive to the optimization conditions present during pre-training. This both opens up directions for more efficient quantization, and poses the question of whether other emergent properties are inherent or can be altered and conditioned by optimization and architecture design choices. We successfully quantize models ranging in size from 410M to 52B with minimal degradation in performance.

Key facts. Cited by 49 papers in the map (data through 13 Oct 2026). Builds on 65 of its 91 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

65 of 91 references.

References · 65

  1. 2017Attention Is All You Needcited by 60,106
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  4. 2020Scaling Laws for Neural Language Modelscited by 6,494
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,836
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,472
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,843
  11. 2022Emergent Abilities of Large Language Modelscited by 2,499
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,979
38 more
  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,978
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,884
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,725
  4. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,585
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,365
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,289
  7. 2017Mixed Precision Trainingcited by 1,147
  8. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,103
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,093
  10. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,058
  11. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,029
  12. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 973
  13. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 841
  14. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 814
  15. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 617
  16. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 617
  17. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 574
  18. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 570
  19. 2016Learning Structured Sparsity in Deep Neural Networkscited by 541
  20. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 522
  21. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 518
  22. 2019The State of Sparsity in Deep Neural Networkscited by 474
  23. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 472
  24. 2021A White Paper on Neural Network Quantizationcited by 467
  25. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 422
  26. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 414
  27. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 399
  28. 2015Deep Learning with Limited Numerical Precisioncited by 372
  29. 2018Generating Wikipedia by Summarizing Long Sequencescited by 335
  30. 2019Q8BERT: Quantized 8Bit BERTcited by 299
  31. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 249
  32. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  33. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 187
  34. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 179
  35. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  36. 2020The Hardware Lotterycited by 160
  37. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  38. 2021The Efficiency Misnomercited by 98

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 8; 2024: 19; 2025: 13; 2026: 9.

49 papers.

Sort:

By area:

  1. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 787
  2. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 379
  3. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 365
  4. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  5. 2024Massive Activations in Large Language Modelscited by 283
  6. 2023Efficient Large Language Models: A Surveycited by 185
  7. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 171
  8. 2024Scaling Laws for Precisioncited by 94
  9. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 85
  10. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  11. 2024Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compressioncited by 56
  12. 2023On the Impact of Calibration Data in Post-training Quantization and Pruningcited by 53
37 more
  1. 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 46
  2. 2024How Does Quantization Affect Multilingual LLMs?cited by 42
  3. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  4. 2026The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinkscited by 26
  5. 2023Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Modelscited by 23
  6. 2026A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Trainingcited by 22
  7. 2024Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMscited by 21
  8. 2024Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scalecited by 21
  9. 2023Training and inference of large language models using 8-bit floating pointcited by 20
  10. 2023Understanding the Effect of Model Compression on Social Bias in Large Language Modelscited by 19
  11. 2024On the Limitations of Compute Thresholds as a Governance Strategycited by 18
  12. 2025Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantizationcited by 14
  13. 2025Training Dynamics Impact Post-Training Quantization Robustnesscited by 14
  14. 2024Outliers and Calibration Sets have Diminishing Effect on Quantization of Modern LLMscited by 12
  15. 2024Fast and Effective Weight Update for Pruned Large Language Modelscited by 11
  16. 2025Are formal and functional linguistic mechanisms dissociated in language models?cited by 11
  17. 2025Fast and Low-Cost Genomic Foundation Models via Outlier Removalcited by 8
  18. 2024HiRE: High Recall Approximate Top-k Estimation for Efficient LLM Inferencecited by 7
  19. 2024LLM See, LLM Do: Guiding Data Generation to Target Non-Differentiable Objectivescited by 6
  20. 2025Thanos: A Block-wise Pruning Algorithm for Efficient Large Language Model Compressioncited by 6
  21. 2026LongAct: Harnessing Intrinsic Activation Patterns for Long-Context Reinforcement Learningcited by 2
  22. 2026A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Modelscited by 2
  23. 2024Is It a Free Lunch for Removing Outliers during Pretraining?cited by 1
  24. 2024NeuroPrune: A Neuro-inspired Topological Sparse Training Algorithm for Large Language Modelscited by 1
  25. 2025NoWag: A Unified Framework for Shape Preserving Compression of Large Language Modelscited by 1
  26. 2025Precision Where It Matters: A Novel Spike Aware Mixed-Precision Quantization Strategy for LLaMA-based Language Modelscited by 1
  27. 2025Z-Pruner: Post-Training Pruning of Large Language Models for Efficiency without Retrainingcited by 1
  28. 2025The Uneven Impact of Post-Training Quantization in Machine Translationcited by 1
  29. 2025Performance Trade-offs of Optimizing Small Language Models for E-Commercecited by 1
  30. 2025TwIST: Rigging the Lottery in Transformers with Independent Subnetwork Trainingcited by 1
  31. 2024Superpipeline: A Universal Approach for Reducing GPU Memory Usage in Large Modelscited by 0
  32. 2024A Data-Free Analytical Quantization Scheme for Deep Learning Modelscited by 0
  33. 2026Flatten The Complex: Joint B-Rep Generation via Compositional k-Cell Particlescited by 0
  34. 2026Elucidating Representation Degradation Problem in Diffusion Model Trainingcited by 0
  35. 2026When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisationcited by 0
  36. 2026The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidationcited by 0
  37. 2026Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling Laws for Precision in High-Dimensional Linear Regression
  2. 2023The Quantization Model of Neural Scaling
  3. 2025Beyond Outliers: A Study of Optimizers Under Quantization
  4. 2023Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Study
  5. 2024Scaling Laws For Mixed Quantization
  6. 2024Art and Science of Quantizing Large-Scale Models: A Comprehensive Overview
  7. 2026The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally
  8. 2026Understanding LLM Quantization through Activation-Guided Compensation and Orthogonal Residuals