# Distilling Object Detectors with Feature Richness

arXiv: https://arxiv.org/abs/2111.00674 · HTML: https://mlsanity.com/paper/2111.00674/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 34 references that are in the map.
- First posted on arXiv 1 Nov 2021.

## Builds on (most cited first, 25 of 25)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Cascade R-CNN: High Quality Object Detection and Instance Segmentation](https://mlsanity.com/paper/1906.09756/) (2019)
- [Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection](https://mlsanity.com/paper/2006.04388/) (2020)
- [A Comprehensive Overhaul of Feature Distillation](https://mlsanity.com/paper/1904.01866/) (2019)
- [Paraphrasing Complex Network: Network Compression via Factor Transfer](https://mlsanity.com/paper/1802.04977/) (2018)
- [Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons](https://mlsanity.com/paper/1811.03233/) (2018)
- [Distilling Object Detectors with Fine-grained Feature Imitation](https://mlsanity.com/paper/1906.03609/) (2019)
- [General Instance Distillation for Object Detection](https://mlsanity.com/paper/2103.02340/) (2021)
- [Distilling Object Detectors via Decoupled Features](https://mlsanity.com/paper/2103.14475/) (2021)
- [Learning Metrics from Teachers: Compact Networks for Image Embedding](https://mlsanity.com/paper/1904.03624/) (2019)
- [Distilling Object Detectors with Task Adaptive Regularization](https://mlsanity.com/paper/2006.13108/) (2020)
- [Graph-based Knowledge Distillation by Multi-head Attention Network](https://mlsanity.com/paper/1907.02226/) (2019)

## Built on by (most cited first, 30 of 38)

- [Masked Generative Distillation](https://mlsanity.com/paper/2205.01529/) (2022)
- [Object-Aware Distillation Pyramid for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2303.05892/) (2023)
- [Knowledge Diffusion for Distillation](https://mlsanity.com/paper/2305.15712/) (2023)
- [CrossKD: Cross-Head Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2306.11369/) (2023)
- [Dense Teacher: Dense Pseudo-Labels for Semi-supervised Object Detection](https://mlsanity.com/paper/2207.02541/) (2022)
- [PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient](https://mlsanity.com/paper/2207.02039/) (2022)
- [FreeKD: Knowledge Distillation via Semantic Frequency Prompt](https://mlsanity.com/paper/2311.12079/) (2023)
- [Masked Distillation with Receptive Tokens](https://mlsanity.com/paper/2205.14589/) (2022)
- [Localization Distillation for Object Detection](https://mlsanity.com/paper/2204.05957/) (2022)
- [Generic-to-Specific Distillation of Masked Autoencoders](https://mlsanity.com/paper/2302.14771/) (2023)
- [ACAM-KD: Adaptive and Cooperative Attention Masking for Knowledge Distillation](https://mlsanity.com/paper/2503.06307/) (2025)
- [Rethinking Centered Kernel Alignment in Knowledge Distillation](https://mlsanity.com/paper/2401.11824/) (2024)
- [Prediction-Guided Distillation for Dense Object Detection](https://mlsanity.com/paper/2203.05469/) (2022)
- [Structural Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2211.13133/) (2022)
- [AMD: Automatic Multi-step Distillation of Large-scale Vision Models](https://mlsanity.com/paper/2407.04208/) (2024)
- [Dual Relation Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2302.05637/) (2023)
- [Avatar Knowledge Distillation: Self-ensemble Teacher Paradigm with Uncertainty](https://mlsanity.com/paper/2305.02722/) (2023)
- [DOT: A Distillation-Oriented Trainer](https://mlsanity.com/paper/2307.08436/) (2023)
- [Let All be Whitened: Multi-teacher Distillation for Efficient Visual Retrieval](https://mlsanity.com/paper/2312.09716/) (2023)
- [Teach-DETR: Better Training DETR with Teachers](https://mlsanity.com/paper/2211.11953/) (2022)
- [A Simple and Generic Framework for Feature Distillation via Channel-wise Transformation](https://mlsanity.com/paper/2303.13212/) (2023)
- [STAL3D: Unsupervised Domain Adaptation for 3D Object Detection via Collaborating Self-Training and Adversarial Learning](https://mlsanity.com/paper/2406.19362/) (2024)
- [Cross-domain Federated Object Detection](https://mlsanity.com/paper/2206.14996/) (2022)
- [Task-Balanced Distillation for Object Detection](https://mlsanity.com/paper/2208.03006/) (2022)
- [Distilling Object Detectors With Global Knowledge](https://mlsanity.com/paper/2210.09022/) (2022)
- [Teaching What You Should Teach: A Data-Based Distillation Method](https://mlsanity.com/paper/2212.05422/) (2022)
- [Knowledge Distillation via Query Selection for Detection Transformer](https://mlsanity.com/paper/2409.06443/) (2024)
- [DMKD: Improving Feature-based Knowledge Distillation for Object Detection Via Dual Masking Augmentation](https://mlsanity.com/paper/2309.02719/) (2023)
- [RdimKD: Generic Distillation Paradigm by Dimensionality Reduction](https://mlsanity.com/paper/2312.08700/) (2023)
- [MonoSKD: General Distillation Framework for Monocular 3D Object Detection via Spearman Correlation Coefficient](https://mlsanity.com/paper/2310.11316/) (2023)

## Abstract (arXiv)

In recent years, large-scale deep models have achieved great success, but the huge computational complexity and massive storage requirements make it a great challenge to deploy them in resource-limited devices. As a model compression and acceleration method, knowledge distillation effectively improves the performance of small models by transferring the dark knowledge from the teacher detector. However, most of the existing distillation-based detection methods mainly imitating features near bounding boxes, which suffer from two limitations. First, they ignore the beneficial features outside the bounding boxes. Second, these methods imitate some features which are mistakenly regarded as the background by the teacher detector. To address the above issues, we propose a novel Feature-Richness Score (FRS) method to choose important features that improve generalized detectability during distilling. The proposed method effectively retrieves the important features outside the bounding boxes and removes the detrimental features within the bounding boxes. Extensive experiments show that our methods achieve excellent performance on both anchor-based and anchor-free detectors. For example, RetinaNet with ResNet-50 achieves 39.7% in mAP on the COCO2017 dataset, which even surpasses the ResNet-101 based teacher detector 38.9% by 0.8%. Our implementation is available at https://github.com/duzhixing/FRS.