# Distilling Object Detectors with Task Adaptive Regularization

arXiv: https://arxiv.org/abs/2006.13108 · HTML: https://mlsanity.com/paper/2006.13108/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jun 2020.

## Built on by (most cited first, 30 of 36)

- [Effective Whole-body Pose Estimation with Two-stages Distillation](https://mlsanity.com/paper/2307.15880/) (2023)
- [Focal and Global Knowledge Distillation for Detectors](https://mlsanity.com/paper/2111.11837/) (2021)
- [General Instance Distillation for Object Detection](https://mlsanity.com/paper/2103.02340/) (2021)
- [Distilling Object Detectors via Decoupled Features](https://mlsanity.com/paper/2103.14475/) (2021)
- [Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation](https://mlsanity.com/paper/2204.02136/) (2022)
- [Localization Distillation for Dense Object Detection](https://mlsanity.com/paper/2102.12252/) (2021)
- [PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient](https://mlsanity.com/paper/2207.02039/) (2022)
- [Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-guided Feature Imitation](https://mlsanity.com/paper/2112.04840/) (2021)
- [Distilling Object Detectors with Feature Richness](https://mlsanity.com/paper/2111.00674/) (2021)
- [DETRDistill: A Universal Knowledge Distillation Framework for DETR-families](https://mlsanity.com/paper/2211.10156/) (2022)
- [G-DetKD: Towards General Distillation Framework for Object Detectors via Contrastive and Semantic-guided Feature Imitation](https://mlsanity.com/paper/2108.07482/) (2021)
- [RadarDistill: Boosting Radar-based Object Detection Performance via Knowledge Distillation from LiDAR Features](https://mlsanity.com/paper/2403.05061/) (2024)
- [Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic Distillation](https://mlsanity.com/paper/2105.12971/) (2021)
- [Masked Distillation with Receptive Tokens](https://mlsanity.com/paper/2205.14589/) (2022)
- [Boosting 3D Object Detection by Simulating Multimodality on Point Clouds](https://mlsanity.com/paper/2206.14971/) (2022)
- [Localization Distillation for Object Detection](https://mlsanity.com/paper/2204.05957/) (2022)
- [Improving Object Detection by Label Assignment Distillation](https://mlsanity.com/paper/2108.10520/) (2021)
- [Prediction-Guided Distillation for Dense Object Detection](https://mlsanity.com/paper/2203.05469/) (2022)
- [Gradient-Guided Knowledge Distillation for Object Detectors](https://mlsanity.com/paper/2303.04240/) (2023)
- [Deep Structured Instance Graph for Distilling Object Detectors](https://mlsanity.com/paper/2109.12862/) (2021)
- [Dual Relation Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2302.05637/) (2023)
- [StereoDistill: Pick the Cream from LiDAR for Distilling Stereo-based 3D Object Detection](https://mlsanity.com/paper/2301.01615/) (2023)
- [Boosting Single-Frame 3D Object Detection by Simulating Multi-Frame Point Clouds](https://mlsanity.com/paper/2207.01030/) (2022)
- [Teach-DETR: Better Training DETR with Teachers](https://mlsanity.com/paper/2211.11953/) (2022)
- [Efficient Object Detection in Optical Remote Sensing Imagery via Attention-based Feature Distillation](https://mlsanity.com/paper/2310.18676/) (2023)
- [Distilling Object Detectors With Global Knowledge](https://mlsanity.com/paper/2210.09022/) (2022)
- [Knowledge Distillation via Query Selection for Detection Transformer](https://mlsanity.com/paper/2409.06443/) (2024)
- [Edge AI-Enabled Chicken Health Detection Based on Enhanced FCOS-Lite and Knowledge Distillation](https://mlsanity.com/paper/2407.09562/) (2024)
- [Representation Disparity-aware Distillation for 3D Object Detection](https://mlsanity.com/paper/2308.10308/) (2023)
- [DiReDi: Distillation and Reverse Distillation for AIoT Applications](https://mlsanity.com/paper/2409.08308/) (2024)

## Abstract (arXiv)

Current state-of-the-art object detectors are at the expense of high computational costs and are hard to deploy to low-end devices. Knowledge distillation, which aims at training a smaller student network by transferring knowledge from a larger teacher model, is one of the promising solutions for model miniaturization. In this paper, we investigate each module of a typical detector in depth, and propose a general distillation framework that adaptively transfers knowledge from teacher to student according to the task specific priors. The intuition is that simply distilling all information from teacher to student is not advisable, instead we should only borrow priors from the teacher model where the student cannot perform well. Towards this goal, we propose a region proposal sharing mechanism to interflow region responses between the teacher and student models. Based on this, we adaptively transfer knowledge at three levels, i.e., feature backbone, classification head, and bounding box regression head, according to which model performs more reasonably. Furthermore, considering that it would introduce optimization dilemma when minimizing distillation loss and detection loss simultaneously, we propose a distillation decay strategy to help improve model generalization via gradually reducing the distillation penalty. Experiments on widely used detection benchmarks demonstrate the effectiveness of our method. In particular, using Faster R-CNN with FPN as an instantiation, we achieve an accuracy of 39.0% with Resnet-50 on COCO dataset, which surpasses the baseline 36.3% by 2.7% points, and even better than the teacher model with 38.5% mAP.