mlsanityData to 7 Oct 2026

Self-Supervised Quantization-Aware Knowledge Distillation

arXiv 2403.11106 ↗First posted 17 Mar 2024cs.LG cs.AI cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Quantization-aware training (QAT) and Knowledge Distillation (KD) are combined to achieve competitive performance in creating low-bit deep learning models. However, existing works applying KD to QAT require tedious hyper-parameter tuning to balance the weights of different loss terms, assume the availability of labeled training data, and require complex, computationally intensive training procedures for good performance. To address these limitations, this paper proposes a novel Self-Supervised Quantization-Aware Knowledge Distillation (SQAKD) framework. SQAKD first unifies the forward and backward dynamics of various quantization functions, making it flexible for incorporating various QAT works. Then it formulates QAT as a co-optimization problem that simultaneously minimizes the KL-Loss between the full-precision and low-bit models for KD and the discretization error for quantization, without supervision from labels. A comprehensive evaluation shows that SQAKD substantially outperforms the state-of-the-art QAT and KD works for a variety of model architectures. Our code is at: https://github.com/kaiqi123/SQAKD.git.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 17 Mar 2024.

Builds on

35 of 51 references.

References · 35

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  4. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  5. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  6. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  7. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  8. 2019Relational Knowledge Distillationcited by 873
  9. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  10. 2019Learned Step Size Quantizationcited by 604
  11. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  12. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
23 more
  1. 2019Similarity-Preserving Knowledge Distillationcited by 507
  2. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  3. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  4. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
  5. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 430
  6. 2019Variational Information Distillation for Knowledge Transfercited by 321
  7. 2019Correlation Congruence for Knowledge Distillationcited by 280
  8. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 272
  9. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 255
  10. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 235
  11. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 234
  12. 2018Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Losscited by 157
  13. 2019Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networkscited by 143
  14. 2019Loss Aware Post-training Quantizationcited by 94
  15. 2021Network Quantization with Element-wise Gradient Scalingcited by 82
  16. 2017Towards Effective Low-bitwidth Convolutional Neural Networkscited by 76
  17. 2019QKD: Quantization-aware Knowledge Distillationcited by 42
  18. 2021MQBench: Towards Reproducible and Deployable Model Quantization Benchmarkcited by 38
  19. 2022Collaborative Multi-Teacher Knowledge Distillation for Learning Low Bit-width Deep Neural Networkscited by 22
  20. 2020Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networkscited by 20
  21. 2023Automatic Attention Pruning: Improving and Automating Model Pruning using Attentionscited by 11
  22. 2021Model compression as constrained optimization, with application to neural nets. Part V: combining compressionscited by 3
  23. 2023A Contrastive Knowledge Transfer Framework for Model Compression and Transfer Learningcited by 2

Built on by

Not cited yet.