mlsanityData to 7 Oct 2026

QGen: On the Ability to Generalize in Quantization Aware Training

arXiv 2404.11769 ↗First posted 17 Apr 2024cs.LG cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Quantization lowers memory usage, computational requirements, and latency by utilizing fewer bits to represent model weights and activations. In this work, we investigate the generalization properties of quantized neural networks, a characteristic that has received little attention despite its implications on model performance. In particular, first, we develop a theoretical model for quantization in neural networks and demonstrate how quantization functions as a form of regularization. Second, motivated by recent work connecting the sharpness of the loss landscape and generalization, we derive an approximate bound for the generalization of quantized models conditioned on the amount of quantization noise. We then validate our hypothesis by experimenting with over 2000 models trained on CIFAR-10, CIFAR-100, and ImageNet datasets on convolutional and transformer-based models.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 44 of its 55 references that are in the map. First posted on arXiv 17 Apr 2024.

Builds on

44 of 55 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  9. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  10. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  11. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  12. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
32 more
  1. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  2. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  3. 2019Learned Step Size Quantizationcited by 604
  4. 2013Network In Networkcited by 600
  5. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  6. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  7. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  8. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  9. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  10. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  11. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  12. 2018Model compression via distillation and quantizationcited by 287
  13. 2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 211
  14. 2017Fisher-Rao Metric, Geometry, and Complexity of Neural Networkscited by 99
  15. 2019Defensive Quantization: When Efficiency Meets Robustnesscited by 95
  16. 2022Sharpness-Aware Training for Freecited by 95
  17. 2020In Search of Robust Measures of Generalizationcited by 79
  18. 2017Training Quantized Nets: A Deeper Understandingcited by 74
  19. 2020Robust Quantization: One Model to Rule Them Allcited by 63
  20. 2020Universally Quantized Neural Compressioncited by 60
  21. 2021Differentiable Model Compression via Pseudo Quantization Noisecited by 44
  22. 2018Alternating Multi-bit Quantization for Recurrent Neural Networkscited by 30
  23. 2017Attacking Binarized Neural Networkscited by 28
  24. 2021Sharpness-aware Quantization for Deep Neural Networkscited by 25
  25. 2021BatchQuant: Quantized-for-all Architecture Search with Robust Quantizercited by 19
  26. 2023Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precisioncited by 17
  27. 2021Quantization of Deep Neural Networks for Accurate Edge Computingcited by 13
  28. 2021On the Adversarial Robustness of Quantized Neural Networkscited by 11
  29. 2022QEBVerif: Quantization Error Bound Verification of Neural Networkscited by 9
  30. 2022SQuAT: Sharpness- and Quantization-Aware Training for BERTcited by 7
  31. 2019Efficient Error-Tolerant Quantized Neural Network Acceleratorscited by 6
  32. 2022Why Quantization Improves Generalization: NTK of Binary Weight Neural Networkscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimatorcited by 4
  2. 2026What Survives When You Compress a Recursive Reasoner for the Edge?cited by 3
  3. 2025FedWSQ: Efficient Federated Learning with Weight Standardization and Distribution-Aware Non-Uniform Quantizationcited by 1
  4. 2025Frequency Composition for Compressed and Domain-Adaptive Neural Networkscited by 0
  5. 2025One-Bit Quantization for Random Features Modelscited by 0
  6. 2026Neural Network Quantization by Learning Low-Loss Subspacescited by 0
  7. 2026Same Predictions, Different Reasons: The Effect of Quantization on Model Explanationscited by 0