# QKD: Quantization-aware Knowledge Distillation

arXiv: https://arxiv.org/abs/1911.12491 · HTML: https://mlsanity.com/paper/1911.12491/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2019.

## Built on by (most cited first, 30 of 44)

- [BiT: Robustly Binarized Multi-distilled Transformer](https://mlsanity.com/paper/2205.13016/) (2022)
- [Dynamic Network Quantization for Efficient Video Inference](https://mlsanity.com/paper/2108.10394/) (2021)
- [Data-Free Knowledge Transfer: A Survey](https://mlsanity.com/paper/2112.15278/) (2021)
- [Collaborative Multi-Teacher Knowledge Distillation for Learning Low Bit-width Deep Neural Networks](https://mlsanity.com/paper/2210.16103/) (2022)
- [Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery](https://mlsanity.com/paper/2601.20088/) (2026)
- [PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation](https://mlsanity.com/paper/2106.14681/) (2021)
- [BatchQuant: Quantized-for-all Architecture Search with Robust Quantizer](https://mlsanity.com/paper/2105.08952/) (2021)
- [Wavelet Feature Maps Compression for Image-to-Image CNNs](https://mlsanity.com/paper/2205.12268/) (2022)
- [KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization](https://mlsanity.com/paper/2101.05938/) (2021)
- [Quantized Feature Distillation for Network Quantization](https://mlsanity.com/paper/2307.10638/) (2023)
- [MetaIE: Distilling a Meta Model from LLM for All Kinds of Information Extraction Tasks](https://mlsanity.com/paper/2404.00457/) (2024)
- [FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation](https://mlsanity.com/paper/2407.07093/) (2024)
- [Poster: Self-Supervised Quantization-Aware Knowledge Distillation](https://mlsanity.com/paper/2309.13220/) (2023)
- [LittleBit: Ultra Low-Bit Quantization via Latent Factorization](https://mlsanity.com/paper/2506.13771/) (2025)
- [ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression](https://mlsanity.com/paper/2106.02241/) (2021)
- [Improving Quantization-aware Training of Low-Precision Network via Block Replacement on Full-Precision Counterpart](https://mlsanity.com/paper/2412.15846/) (2024)
- [Improved Techniques for Quantizing Deep Networks with Adaptive Bit-Widths](https://mlsanity.com/paper/2103.01435/) (2021)
- [Efficient Speech Representation Learning with Low-Bit Quantization](https://mlsanity.com/paper/2301.00652/) (2022)
- [Feature Affinity Assisted Knowledge Distillation and Quantization of Deep Neural Networks on Label-Free Data](https://mlsanity.com/paper/2302.10899/) (2023)
- [TSAK: Two-Stage Semantic-Aware Knowledge Distillation for Efficient Wearable Modality and Model Optimization in Manufacturing Lines](https://mlsanity.com/paper/2408.14146/) (2024)
- [Bimodal Distributed Binarized Neural Networks](https://mlsanity.com/paper/2204.02004/) (2022)
- [Vertical Layering of Quantized Neural Networks for Heterogeneous Inference](https://mlsanity.com/paper/2212.05326/) (2022)
- [Robust Training of Neural Networks at Arbitrary Precision and Sparsity](https://mlsanity.com/paper/2409.09245/) (2024)
- [Training Multi-bit Quantized and Binarized Networks with A Learnable Symmetric Quantizer](https://mlsanity.com/paper/2104.00210/) (2021)
- [Efficient and Robust Quantization-aware Training via Adaptive Coreset Selection](https://mlsanity.com/paper/2306.07215/) (2023)
- [Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models](https://mlsanity.com/paper/2307.05972/) (2023)
- [Arch-Net: Model Distillation for Architecture Agnostic Model Deployment](https://mlsanity.com/paper/2111.01135/) (2021)
- [MobileNMT: Enabling Translation in 15MB and 30ms](https://mlsanity.com/paper/2306.04235/) (2023)
- [On the Impact of White-box Deployment Strategies for Edge AI on Latency and Model Performance](https://mlsanity.com/paper/2411.00907/) (2024)
- [Binary Neural Networks for Large Language Model: A Survey](https://mlsanity.com/paper/2502.19008/) (2025)

## Abstract (arXiv)

Quantization and Knowledge distillation (KD) methods are widely used to reduce memory and power consumption of deep neural networks (DNNs), especially for resource-constrained edge devices. Although their combination is quite promising to meet these requirements, it may not work as desired. It is mainly because the regularization effect of KD further diminishes the already reduced representation power of a quantized model. To address this short-coming, we propose Quantization-aware Knowledge Distillation (QKD) wherein quantization and KD are care-fully coordinated in three phases. First, Self-studying (SS) phase fine-tunes a quantized low-precision student network without KD to obtain a good initialization. Second, Co-studying (CS) phase tries to train a teacher to make it more quantizaion-friendly and powerful than a fixed teacher. Finally, Tutoring (TU) phase transfers knowledge from the trained teacher to the student. We extensively evaluate our method on ImageNet and CIFAR-10/100 datasets and show an ablation study on networks with both standard and depthwise-separable convolutions. The proposed QKD outperformed existing state-of-the-art methods (e.g., 1.3% improvement on ResNet-18 with W4A4, 2.6% on MobileNetV2 with W4A4). Additionally, QKD could recover the full-precision accuracy at as low as W3A3 quantization on ResNet and W6A6 quantization on MobilenetV2.