# Learning Efficient Detector with Semi-supervised Adaptive Distillation

arXiv: https://arxiv.org/abs/1901.00366 · HTML: https://mlsanity.com/paper/1901.00366/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jan 2019.

## Built on by (most cited first, 10 of 10)

- [Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective](https://mlsanity.com/paper/2102.00650/) (2021)
- [Distilling Object Detectors via Decoupled Features](https://mlsanity.com/paper/2103.14475/) (2021)
- [Teacher's pet: understanding and mitigating biases in distillation](https://mlsanity.com/paper/2106.10494/) (2021)
- [Dynamic Contrastive Distillation for Image-Text Retrieval](https://mlsanity.com/paper/2207.01426/) (2022)
- [WSSOD: A New Pipeline for Weakly- and Semi-Supervised Object Detection](https://mlsanity.com/paper/2105.11293/) (2021)
- [Update Compression for Deep Neural Networks on the Edge](https://mlsanity.com/paper/2203.04516/) (2022)
- [AMD: Adaptive Masked Distillation for Object Detection](https://mlsanity.com/paper/2301.13538/) (2023)
- [Beyond Classification: Knowledge Distillation using Multi-Object Impressions](https://mlsanity.com/paper/2110.14215/) (2021)
- [Adaptive Self-Distillation for Minimizing Client Drift in Heterogeneous Federated Learning](https://mlsanity.com/paper/2305.19600/) (2023)
- [Distillation from heterogeneous unlabeled collections](https://mlsanity.com/paper/2201.06507/) (2022)

## Abstract (arXiv)

Knowledge Distillation (KD) has been used in image classification for model compression. However, rare studies apply this technology on single-stage object detectors. Focal loss shows that the accumulated errors of easily-classified samples dominate the overall loss in the training process. This problem is also encountered when applying KD in the detection task. For KD, the teacher-defined hard samples are far more important than any others. We propose ADL to address this issue by adaptively mimicking the teacher's logits, with more attention paid on two types of hard samples: hard-to-learn samples predicted by teacher with low certainty and hard-to-mimic samples with a large gap between the teacher's and the student's prediction. ADL enlarges the distillation loss for hard-to-learn and hard-to-mimic samples and reduces distillation loss for the dominant easy samples, enabling distillation to work on the single-stage detector first time, even if the student and the teacher are identical. Besides, ADL is effective in both the supervised setting and the semi-supervised setting, even when the labeled data and unlabeled data are from different distributions. For distillation on unlabeled data, ADL achieves better performance than existing data distillation which simply utilizes hard targets, making the student detector surpass its teacher. On the COCO database, semi-supervised adaptive distillation (SAD) makes a student detector with a backbone of ResNet-50 surpasses its teacher with a backbone of ResNet-101, while the student has half of the teacher's computation complexity. The code is avaiable at https://github.com/Tangshitao/Semi-supervised-Adaptive-Distillation