mlsanityData to 14 Oct 2026

Efficiently Training A Flat Neural Network Before It has been Quantizated

arXiv 2511.01462 ↗First posted 3 Nov 2025cs.CV cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Post-training quantization (PTQ) for vision transformers (ViTs) has garnered significant attention due to its efficiency in compressing models. However, existing methods typically overlook the relationship between a well-trained NN and the quantized model, leading to considerable quantization error for PTQ. However, it is unclear how to efficiently train a model-agnostic neural network which is tailored for a predefined precision low-bit model. In this paper, we firstly discover that a flat full precision neural network is crucial for low-bit quantization. To achieve this, we propose a framework that proactively pre-conditions the model by measuring and disentangling the error sources. Specifically, both the Activation Quantization Error (AQE) and the Weight Quantization Error (WQE) are statistically modeled as independent Gaussian noises. We study several noise injection optimization methods to obtain a flat minimum. Experimental results attest to the effectiveness of our approach. These results open novel pathways for obtaining low-bit PTQ models.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 28 of its 31 references that are in the map. First posted on arXiv 3 Nov 2025.

Builds on

28 of 31 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  4. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  9. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  10. 2019Learned Step Size Quantizationcited by 628
  11. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  12. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 560
16 more
  1. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
  2. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  3. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  4. 2017How to Escape Saddle Points Efficientlycited by 285
  5. 2023Q-Diffusion: Quantizing Diffusion Modelscited by 273
  6. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
  7. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 189
  8. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 180
  9. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 119
  10. 2022PD-Quant: Post-Training Quantization based on Prediction Difference Metriccited by 73
  11. 2022PSAQ-ViT V2: Towards Accurate and General Data-Free Quantization for Vision Transformerscited by 42
  12. 2021MQBench: Towards Reproducible and Deployable Model Quantization Benchmarkcited by 40
  13. 2024BitsFusion: 1.99 bits Weight Quantization of Diffusion Modelcited by 37
  14. 2022NIPQ: Noise proxy-based Integrated Pseudo-Quantizationcited by 35
  15. 2023Solving Oscillation Problem in Post-Training Quantization Through a Theoretical Perspectivecited by 17
  16. 2023Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precisioncited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Neural Network Quantization by Learning Low-Loss Subspacescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Accurate Post-Training Quantization of Vision Transformers via Error Reduction
  2. 2025AIQViT: Architecture-Informed Post-Training Quantization for Vision Transformers
  3. 2024COMQ: A Backpropagation-Free Algorithm for Post-Training Quantization
  4. 2024MPTQ-ViT: Mixed-Precision Post-Training Quantization for Vision Transformer
  5. 2021PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantization
  6. 2025APHQ-ViT: Post-Training Quantization with Average Perturbation Hessian Based Reconstruction for Vision Transformers
  7. 2025IPTQ-ViT: Post-Training Quantization of Non-linear Functions for Integer-only Vision Transformers
  8. 2022RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers