mlsanityData to 14 Oct 2026

Quantized Feature Distillation for Network Quantization

arXiv 2307.10638 ↗First posted 20 Jul 2023cs.CVcited by 15

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Neural network quantization aims to accelerate and trim full-precision neural network models by using low bit approximations. Methods adopting the quantization aware training (QAT) paradigm have recently seen a rapid growth, but are often conceptually complicated. This paper proposes a novel and highly effective QAT method, quantized feature distillation (QFD). QFD first trains a quantized (or binarized) representation as the teacher, then quantize the network using knowledge distillation (KD). Quantitative results show that QFD is more flexible and effective (i.e., quantization friendly) than previous quantization methods. QFD surpasses existing methods by a noticeable margin on not only image classification but also object detection, albeit being much simpler. Furthermore, QFD quantizes ViT and Swin-Transformer on MS-COCO detection and segmentation, which verifies its potential in real world deployment. To the best of our knowledge, this is the first time that vision transformers have been quantized in object detection and image segmentation tasks.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 42 of its 48 references that are in the map. First posted on arXiv 20 Jul 2023.

Builds on

42 of 48 references.

References · 42

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2016Layer Normalizationcited by 5,669
  8. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  10. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  11. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  12. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
30 more
  1. 2019Learned Step Size Quantizationcited by 628
  2. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  3. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  4. 2021A White Paper on Neural Network Quantizationcited by 472
  5. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  6. 2018Model compression via distillation and quantizationcited by 289
  7. 2021Post-Training Quantization for Vision Transformercited by 278
  8. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 274
  9. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 238
  10. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 189
  11. 2018Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Losscited by 160
  12. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformercited by 151
  13. 2019Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networkscited by 148
  14. 2021Focal and Global Knowledge Distillation for Detectorscited by 142
  15. 2017Deep Learning with Low Precision by Half-wave Gaussian Quantizationcited by 117
  16. 2021Distilling Object Detectors via Decoupled Featurescited by 106
  17. 2017Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracycited by 101
  18. 2021Network Quantization with Element-wise Gradient Scalingcited by 87
  19. 2021Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimationcited by 79
  20. 2021Distilling Virtual Examples for Long-tailed Recognitioncited by 72
  21. 2019QKD: Quantization-aware Knowledge Distillationcited by 44
  22. 2021Learnable Companding Quantization for Accurate Low-bit Neural Networkscited by 42
  23. 2021MQBench: Towards Reproducible and Deployable Model Quantization Benchmarkcited by 40
  24. 2019Training Quantized Neural Networks with a Full-precision Auxiliary Modulecited by 35
  25. 2022VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformercited by 27
  26. 2020Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networkscited by 21
  27. 2021Distance-aware Quantizationcited by 19
  28. 2021Cluster-Promoting Quantization with Bit-Drop for Minimizing Network Quantization Losscited by 12
  29. 2020Switchable Precision Neural Networkscited by 6
  30. 2018Learning Effective Binary Visual Representations with Deep Networkscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 6; 2026: 4.

15 papers.

Sort:

By area:

  1. 2024Quantization without Tearscited by 25
  2. 2025Low-bit Model Quantization for Deep Neural Networks: A Surveycited by 11
  3. 2024Enhancing Descriptive Captions with Visual Attributes for Multimodal Perceptioncited by 10
  4. 2025GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformerscited by 8
  5. 2024All You Need in Knowledge Distillation Is a Tailored Coordinate Systemcited by 6
  6. 2024Improving Quantization-aware Training of Low-Precision Network via Block Replacement on Full-Precision Counterpartcited by 6
  7. 2024Rectify the Regression Bias in Long-Tailed Object Detectioncited by 5
  8. 2025QwT-v2: Practical, Effective and Efficient Post-Training Quantizationcited by 5
  9. 2025Energy-Efficient Transformer Inference: Optimization Strategies for Time Series Classificationcited by 2
  10. 2025Punching Above Precision: Small Quantized Model Distillation with Learnable Regularizercited by 2
  11. 2026Decoder-Free Distillation for Quantized Image Restorationcited by 1
  12. 2026SQuaT: Self-Supervised Knowledge Distillation via Student-Aware Quantized Teacher Featurescited by 1
3 more
  1. 2025Q2: Quantization-Aware Gradient Balancing and Attention Alignment for Low-Bit Quantizationcited by 0
  2. 2026QATMA: Quantization-Aware Training with Multimodal Alignment for Open-Vocabulary Object Detectioncited by 0
  3. 2026Progressive Multi-Ancestor Bit-Depth Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Data-Augmented Quantization-Aware Knowledge Distillation
  2. 2024Self-Supervised Quantization-Aware Knowledge Distillation
  3. 2023Poster: Self-Supervised Quantization-Aware Knowledge Distillation
  4. 2023Bag of Tricks with Quantized Convolutional Neural Networks for image classification
  5. 2025Architectural Insights into Knowledge Distillation for Object Detection: A Comprehensive Review
  6. 2023Feature Affinity Assisted Knowledge Distillation and Quantization of Deep Neural Networks on Label-Free Data
  7. 2022A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification
  8. 2023QD-BEV : Quantization-aware View-guided Distillation for Multi-view 3D Object Detection