# Quantized Feature Distillation for Network Quantization

arXiv: https://arxiv.org/abs/2307.10638 · HTML: https://mlsanity.com/paper/2307.10638/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 42 of its 48 references that are in the map.
- First posted on arXiv 20 Jul 2023.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 30 of 42)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [Learned Step Size Quantization](https://mlsanity.com/paper/1902.08153/) (2019)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [Post-Training Quantization for Vision Transformer](https://mlsanity.com/paper/2106.14156/) (2021)
- [LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks](https://mlsanity.com/paper/1807.10029/) (2018)
- [Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks](https://mlsanity.com/paper/1908.05033/) (2019)
- [LSQ+: Improving low-bit quantization through learnable offsets and better initialization](https://mlsanity.com/paper/2004.09576/) (2020)
- [Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Loss](https://mlsanity.com/paper/1808.05779/) (2018)
- [FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer](https://mlsanity.com/paper/2111.13824/) (2021)
- [Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks](https://mlsanity.com/paper/1909.13144/) (2019)
- [Focal and Global Knowledge Distillation for Detectors](https://mlsanity.com/paper/2111.11837/) (2021)
- [Deep Learning with Low Precision by Half-wave Gaussian Quantization](https://mlsanity.com/paper/1702.00953/) (2017)
- [Distilling Object Detectors via Decoupled Features](https://mlsanity.com/paper/2103.14475/) (2021)
- [Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy](https://mlsanity.com/paper/1711.05852/) (2017)
- [Network Quantization with Element-wise Gradient Scaling](https://mlsanity.com/paper/2104.00903/) (2021)

## Built on by (most cited first, 15 of 15)

- [Quantization without Tears](https://mlsanity.com/paper/2411.13918/) (2024)
- [Low-bit Model Quantization for Deep Neural Networks: A Survey](https://mlsanity.com/paper/2505.05530/) (2025)
- [Enhancing Descriptive Captions with Visual Attributes for Multimodal Perception](https://mlsanity.com/paper/2412.14233/) (2024)
- [GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers](https://mlsanity.com/paper/2506.11784/) (2025)
- [All You Need in Knowledge Distillation Is a Tailored Coordinate System](https://mlsanity.com/paper/2412.09388/) (2024)
- [Improving Quantization-aware Training of Low-Precision Network via Block Replacement on Full-Precision Counterpart](https://mlsanity.com/paper/2412.15846/) (2024)
- [Rectify the Regression Bias in Long-Tailed Object Detection](https://mlsanity.com/paper/2401.15885/) (2024)
- [QwT-v2: Practical, Effective and Efficient Post-Training Quantization](https://mlsanity.com/paper/2505.20932/) (2025)
- [Energy-Efficient Transformer Inference: Optimization Strategies for Time Series Classification](https://mlsanity.com/paper/2502.16627/) (2025)
- [Punching Above Precision: Small Quantized Model Distillation with Learnable Regularizer](https://mlsanity.com/paper/2509.20854/) (2025)
- [Decoder-Free Distillation for Quantized Image Restoration](https://mlsanity.com/paper/2603.09624/) (2026)
- [SQuaT: Self-Supervised Knowledge Distillation via Student-Aware Quantized Teacher Features](https://mlsanity.com/paper/2608.10709/) (2026)
- [Q2: Quantization-Aware Gradient Balancing and Attention Alignment for Low-Bit Quantization](https://mlsanity.com/paper/2511.05898/) (2025)
- [QATMA: Quantization-Aware Training with Multimodal Alignment for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2603.05964/) (2026)
- [Progressive Multi-Ancestor Bit-Depth Distillation](https://mlsanity.com/paper/2610.04100/) (2026)

## Abstract (arXiv)

Neural network quantization aims to accelerate and trim full-precision neural network models by using low bit approximations. Methods adopting the quantization aware training (QAT) paradigm have recently seen a rapid growth, but are often conceptually complicated. This paper proposes a novel and highly effective QAT method, quantized feature distillation (QFD). QFD first trains a quantized (or binarized) representation as the teacher, then quantize the network using knowledge distillation (KD). Quantitative results show that QFD is more flexible and effective (i.e., quantization friendly) than previous quantization methods. QFD surpasses existing methods by a noticeable margin on not only image classification but also object detection, albeit being much simpler. Furthermore, QFD quantizes ViT and Swin-Transformer on MS-COCO detection and segmentation, which verifies its potential in real world deployment. To the best of our knowledge, this is the first time that vision transformers have been quantized in object detection and image segmentation tasks.