# Distilling Object Detectors via Decoupled Features

arXiv: https://arxiv.org/abs/2103.14475 · HTML: https://mlsanity.com/paper/2103.14475/

## Key facts

- Cited by 106 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 55 of its 71 references that are in the map.
- First posted on arXiv 26 Mar 2021.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 30 of 55)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [GhostNet: More Features from Cheap Operations](https://mlsanity.com/paper/1911.11907/) (2019)
- [A Comprehensive Overhaul of Feature Distillation](https://mlsanity.com/paper/1904.01866/) (2019)
- [Variational Information Distillation for Knowledge Transfer](https://mlsanity.com/paper/1904.05835/) (2019)
- [Relation Networks for Object Detection](https://mlsanity.com/paper/1711.11575/) (2017)
- [Paraphrasing Complex Network: Network Compression via Factor Transfer](https://mlsanity.com/paper/1802.04977/) (2018)
- [Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons](https://mlsanity.com/paper/1811.03233/) (2018)
- [Dynamic Network Surgery for Efficient DNNs](https://mlsanity.com/paper/1608.04493/) (2016)
- [Quantized Convolutional Neural Networks for Mobile Devices](https://mlsanity.com/paper/1512.06473/) (2015)
- [Learning Deep Representations with Probabilistic Knowledge Transfer](https://mlsanity.com/paper/1803.10837/) (2018)

## Built on by (most cited first, 30 of 106)

- [Focal and Global Knowledge Distillation for Detectors](https://mlsanity.com/paper/2111.11837/) (2021)
- [Masked Generative Distillation](https://mlsanity.com/paper/2205.01529/) (2022)
- [One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation](https://mlsanity.com/paper/2310.19444/) (2023)
- [D-FINE: Redefine Regression Task in DETRs as Fine-grained Distribution Refinement](https://mlsanity.com/paper/2410.13842/) (2024)
- [Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation](https://mlsanity.com/paper/2204.02136/) (2022)
- [BEVDistill: Cross-Modal BEV Distillation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2211.09386/) (2022)
- [Object-Aware Distillation Pyramid for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2303.05892/) (2023)
- [Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation](https://mlsanity.com/paper/2107.01378/) (2021)
- [MixFormerV2: Efficient Fully Transformer Tracking](https://mlsanity.com/paper/2305.15896/) (2023)
- [LiDAR Distillation: Bridging the Beam-Induced Domain Gap for 3D Object Detection](https://mlsanity.com/paper/2203.14956/) (2022)
- [Knowledge Diffusion for Distillation](https://mlsanity.com/paper/2305.15712/) (2023)
- [CrossKD: Cross-Head Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2306.11369/) (2023)
- [Gold-YOLO: Efficient Object Detector via Gather-and-Distribute Mechanism](https://mlsanity.com/paper/2309.11331/) (2023)
- [UniDistill: A Universal Cross-Modality Knowledge Distillation Framework for 3D Object Detection in Bird's-Eye View](https://mlsanity.com/paper/2303.15083/) (2023)
- [Instance-Conditional Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2110.12724/) (2021)
- [Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillation](https://mlsanity.com/paper/2206.06067/) (2022)
- [PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient](https://mlsanity.com/paper/2207.02039/) (2022)
- [Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-guided Feature Imitation](https://mlsanity.com/paper/2112.04840/) (2021)
- [Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI](https://mlsanity.com/paper/2111.06061/) (2021)
- [Distilling Object Detectors with Feature Richness](https://mlsanity.com/paper/2111.00674/) (2021)
- [DETRDistill: A Universal Knowledge Distillation Framework for DETR-families](https://mlsanity.com/paper/2211.10156/) (2022)
- [Visual Tuning](https://mlsanity.com/paper/2305.06061/) (2023)
- [PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D Detection](https://mlsanity.com/paper/2205.11098/) (2022)
- [TinySAM: Pushing the Envelope for Efficient Segment Anything Model](https://mlsanity.com/paper/2312.13789/) (2023)
- [RadarDistill: Boosting Radar-based Object Detection Performance via Knowledge Distillation from LiDAR Features](https://mlsanity.com/paper/2403.05061/) (2024)
- [RadOcc: Learning Cross-Modality Occupancy Knowledge through Rendering Assisted Distillation](https://mlsanity.com/paper/2312.11829/) (2023)
- [Masked Distillation with Receptive Tokens](https://mlsanity.com/paper/2205.14589/) (2022)
- [VanillaNet: the Power of Minimalism in Deep Learning](https://mlsanity.com/paper/2305.12972/) (2023)
- [Boosting 3D Object Detection by Simulating Multimodality on Point Clouds](https://mlsanity.com/paper/2206.14971/) (2022)
- [LGD: Label-guided Self-distillation for Object Detection](https://mlsanity.com/paper/2109.11496/) (2021)

## Abstract (arXiv)

Knowledge distillation is a widely used paradigm for inheriting information from a complicated teacher network to a compact student network and maintaining the strong performance. Different from image classification, object detectors are much more sophisticated with multiple loss functions in which features that semantic information rely on are tangled. In this paper, we point out that the information of features derived from regions excluding objects are also essential for distilling the student detector, which is usually ignored in existing approaches. In addition, we elucidate that features from different regions should be assigned with different importance during distillation. To this end, we present a novel distillation algorithm via decoupled features (DeFeat) for learning a better student detector. Specifically, two levels of decoupled features will be processed for embedding useful information into the student, i.e., decoupled features from neck and decoupled proposals from classification head. Extensive experiments on various detectors with different backbones show that the proposed DeFeat is able to surpass the state-of-the-art distillation methods for object detection. For example, DeFeat improves ResNet50 based Faster R-CNN from 37.4% to 40.9% mAP, and improves ResNet50 based RetinaNet from 36.5% to 39.7% mAP on COCO benchmark. Our implementation is available at https://github.com/ggjy/DeFeat.pytorch.