mlsanityData to 14 Oct 2026

Poster: Self-Supervised Quantization-Aware Knowledge Distillation

arXiv 2309.13220 ↗First posted 22 Sep 2023cs.CV cs.AIcited by 11

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Quantization-aware training (QAT) starts with a pre-trained full-precision model and performs quantization during retraining. However, existing QAT works require supervision from the labels and they suffer from accuracy loss due to reduced precision. To address these limitations, this paper proposes a novel Self-Supervised Quantization-Aware Knowledge Distillation framework (SQAKD). SQAKD first unifies the forward and backward dynamics of various quantization functions and then reframes QAT as a co-optimization problem that simultaneously minimizes the KL-Loss and the discretization error, in a self-supervised manner. The evaluation shows that SQAKD significantly improves the performance of various state-of-the-art QAT works. SQAKD establishes stronger baselines and does not require extensive labeled training data, potentially making state-of-the-art QAT research more accessible.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 35 of its 51 references that are in the map. First posted on arXiv 22 Sep 2023.

Builds on

35 of 51 references.

References · 35

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  4. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  5. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  6. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  7. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
  8. 2019Relational Knowledge Distillationcited by 889
  9. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  10. 2019Learned Step Size Quantizationcited by 628
  11. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  12. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
23 more
  1. 2019Similarity-Preserving Knowledge Distillationcited by 516
  2. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  3. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  4. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  5. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 431
  6. 2019Variational Information Distillation for Knowledge Transfercited by 324
  7. 2019Correlation Congruence for Knowledge Distillationcited by 285
  8. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 274
  9. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 260
  10. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 238
  11. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 238
  12. 2018Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Losscited by 160
  13. 2019Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networkscited by 148
  14. 2019Loss Aware Post-training Quantizationcited by 97
  15. 2021Network Quantization with Element-wise Gradient Scalingcited by 87
  16. 2017Towards Effective Low-bitwidth Convolutional Neural Networkscited by 78
  17. 2019QKD: Quantization-aware Knowledge Distillationcited by 44
  18. 2021MQBench: Towards Reproducible and Deployable Model Quantization Benchmarkcited by 40
  19. 2022Collaborative Multi-Teacher Knowledge Distillation for Learning Low Bit-width Deep Neural Networkscited by 23
  20. 2020Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networkscited by 21
  21. 2023Automatic Attention Pruning: Improving and Automating Model Pruning using Attentionscited by 11
  22. 2021Model compression as constrained optimization, with application to neural nets. Part V: combining compressionscited by 3
  23. 2023A Contrastive Knowledge Transfer Framework for Model Compression and Transfer Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 6.

11 papers.

  1. 2025Saliency-Aware Quantized Imitation Learning for Efficient Robotic Controlcited by 12
  2. 2026Gated Relational Alignment via Confidence-based Distillation for Efficient VLMscited by 6
  3. 2025Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flashcited by 4
  4. 2025Punching Above Precision: Small Quantized Model Distillation with Learnable Regularizercited by 2
  5. 2024Advanced Knowledge Transfer: Refined Feature Distillation for Zero-Shot Quantization in Edge Computingcited by 1
  6. 2025Data-Augmented Quantization-Aware Knowledge Distillationcited by 1
  7. 2026Decoder-Free Distillation for Quantized Image Restorationcited by 1
  8. 2026SQuaT: Self-Supervised Knowledge Distillation via Student-Aware Quantized Teacher Featurescited by 1
  9. 2026Nano-U: Efficient Terrain Segmentation for Tiny Robot Navigationcited by 0
  10. 2026FTerViT: Fully Ternary Vision Transformercited by 0
  11. 2026Progressive Multi-Ancestor Bit-Depth Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Self-Supervised Quantization-Aware Knowledge Distillation
  2. 2026A Target-Centric Survey of Quantization-Aware Training
  3. 2026What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Study
  4. 2022Synergistic Self-supervised and Quantization Learning
  5. 2024In-Distribution Consistency Regularization Improves the Generalization of Quantization-Aware Training
  6. 2026StableQAT: Stable Quantization-Aware Training at Ultra-Low Bitwidths
  7. 2026Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery
  8. 2024Oh! We Freeze: Improving Quantized Knowledge Distillation via Signal Propagation Analysis for Large Language Models