In-Distribution Consistency Regularization Improves the Generalization of Quantization-Aware Training
Not on the map. Why?
Abstract (arXiv)
Although existing Quantization-Aware Training (QAT) methods intensively depend on knowledge distillation to guarantee performance, QAT still suffers from severe performance drop. The experiments have shown that vanilla quantization is sensitive to the perturbation from both the input and weights. Therefore, we assume that the generalization ability of QAT is predominantly caused by both the intrinsic instability (training time) and the limited generalization ability (testing time). In this paper, we address both issues from a new perspective by leveraging Consistency Regularization (CR) to improve the generalization ability of QAT. Empirical results and theoretical analysis verify that CR would bring a good generalization ability to different network architectures and various QAT methods. Extensive experiments demonstrate that our approach significantly outperforms current state-of-the-art QAT methods and even the FP counterparts. On CIFAR-10, the proposed method improves by 3.79% compared to the baseline method using ResNet18, and improves by 3.84% compared to the baseline method using the lightweight model MobileNet.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 44 of its 55 references that are in the map. First posted on arXiv 21 Feb 2024.
Builds on
44 of 55 references.
References · 44
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
- 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
- 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
- 2020In Search of Lost Domain Generalizationcited by 1,042
32 more
- 2020Designing Network Design Spacescited by 954
- 2019Learned Step Size Quantizationcited by 604
- 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
- 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
- 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
- 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
- 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
- 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 288
- 2018Model compression via distillation and quantizationcited by 287
- 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 234
- 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 180
- 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 175
- 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 116
- 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 106
- 2019Defensive Quantization: When Efficiency Meets Robustnesscited by 95
- 2021Network Quantization with Element-wise Gradient Scalingcited by 82
- 2019Towards Large yet Imperceptible Adversarial Image Perturbations with Perceptual Color Distancecited by 72
- 2019AdaBits: Neural Network Quantization with Adaptive Bit-Widthscited by 63
- 2020Robust Quantization: One Model to Rule Them Allcited by 63
- 2021Improved Regularization and Robustness for Fine-tuning in Neural Networkscited by 45
- 2021Differentiable Model Compression via Pseudo Quantization Noisecited by 44
- 2022PSAQ-ViT V2: Towards Accurate and General Data-Free Quantization for Vision Transformerscited by 42
- 2019SWALP : Stochastic Weight Averaging in Low-Precision Trainingcited by 40
- 2019The intriguing role of module criticality in the generalization of deep networkscited by 40
- 2018Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?cited by 38
- 2021MQBench: Towards Reproducible and Deployable Model Quantization Benchmarkcited by 38
- 2020Once Quantization-Aware Training: High Performance Extremely Low-bit Architecture Searchcited by 26
- 2022Shapley-NAS: Discovering Operation Contribution for Neural Architecture Searchcited by 23
- 2023Q-DETR: An Efficient Low-Bit Quantized Detection Transformercited by 23
- 2018Quantization Mimic: Towards Very Tiny CNN for Object Detectioncited by 20
- 2022IDa-Det: An Information Discrepancy-aware Distillation for 1-bit Detectorscited by 12
- 2022Synergistic Self-supervised and Quantization Learningcited by 10
Built on by
Citing papers per year: 2025: 1.
1 paper.