# Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-guided Feature Imitation

arXiv: https://arxiv.org/abs/2112.04840 · HTML: https://mlsanity.com/paper/2112.04840/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 20 of its 24 references that are in the map.
- First posted on arXiv 9 Dec 2021.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection](https://mlsanity.com/paper/1912.02424/) (2019)
- [Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection](https://mlsanity.com/paper/2006.04388/) (2020)
- [VarifocalNet: An IoU-aware Dense Object Detector](https://mlsanity.com/paper/2008.13367/) (2020)
- [Distilling Object Detectors with Fine-grained Feature Imitation](https://mlsanity.com/paper/1906.03609/) (2019)
- [General Instance Distillation for Object Detection](https://mlsanity.com/paper/2103.02340/) (2021)
- [Distilling Object Detectors via Decoupled Features](https://mlsanity.com/paper/2103.14475/) (2021)
- [Generalized Focal Loss V2: Learning Reliable Localization Quality Estimation for Dense Object Detection](https://mlsanity.com/paper/2011.12885/) (2020)
- [Distilling Object Detectors with Task Adaptive Regularization](https://mlsanity.com/paper/2006.13108/) (2020)

## Built on by (most cited first, 30 of 40)

- [Curriculum Temperature for Knowledge Distillation](https://mlsanity.com/paper/2211.16231/) (2022)
- [Logit Standardization in Knowledge Distillation](https://mlsanity.com/paper/2403.01427/) (2024)
- [CrossKD: Cross-Head Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2306.11369/) (2023)
- [PseCo: Pseudo Labeling and Consistency Training for Semi-Supervised Object Detection](https://mlsanity.com/paper/2203.16317/) (2022)
- [PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient](https://mlsanity.com/paper/2207.02039/) (2022)
- [ViTKD: Practical Guidelines for ViT feature knowledge distillation](https://mlsanity.com/paper/2209.02432/) (2022)
- [Localization Distillation for Object Detection](https://mlsanity.com/paper/2204.05957/) (2022)
- [DTG-SSOD: Dense Teacher Guidance for Semi-Supervised Object Detection](https://mlsanity.com/paper/2207.05536/) (2022)
- [Teach-DETR: Better Training DETR with Teachers](https://mlsanity.com/paper/2211.11953/) (2022)
- [A Simple and Generic Framework for Feature Distillation via Channel-wise Transformation](https://mlsanity.com/paper/2303.13212/) (2023)
- [Bridging Cross-task Protocol Inconsistency for Distillation in Dense Object Detection](https://mlsanity.com/paper/2308.14286/) (2023)
- [Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2504.13825/) (2025)
- [Task-Balanced Distillation for Object Detection](https://mlsanity.com/paper/2208.03006/) (2022)
- [Distilling Object Detectors With Global Knowledge](https://mlsanity.com/paper/2210.09022/) (2022)
- [Adversarial Attacks and Defenses in Machine Learning-Powered Networks: A Contemporary Survey](https://mlsanity.com/paper/2303.06302/) (2023)
- [Aligning in a Compact Space: Contrastive Knowledge Distillation between Heterogeneous Architectures](https://mlsanity.com/paper/2405.18524/) (2024)
- [Knowledge Distillation via Query Selection for Detection Transformer](https://mlsanity.com/paper/2409.06443/) (2024)
- [Random Conditioning with Distillation for Data-Efficient Diffusion Model Compression](https://mlsanity.com/paper/2504.02011/) (2025)
- [Cyclic-Bootstrap Labeling for Weakly Supervised Object Detection](https://mlsanity.com/paper/2308.05991/) (2023)
- [Positive-Unlabeled Reinforcement Learning Distillation for On-Premise Small Models](https://mlsanity.com/paper/2601.20687/) (2026)
- [Enhancing Indoor Occupancy Prediction via Sparse Query-Based Multi-Level Consistent Knowledge Distillation](https://mlsanity.com/paper/2602.02318/) (2026)
- [LumiNet: Perception-Driven Knowledge Distillation via Statistical Logit Calibration](https://mlsanity.com/paper/2310.03669/) (2023)
- [RdimKD: Generic Distillation Paradigm by Dimensionality Reduction](https://mlsanity.com/paper/2312.08700/) (2023)
- [Why does Knowledge Distillation Work? Rethink its Attention and Fidelity Mechanism](https://mlsanity.com/paper/2405.00739/) (2024)
- [Enhancing Logits Distillation with Plug&Play Kendall's τ Ranking Loss](https://mlsanity.com/paper/2409.17823/) (2024)
- [Smooth and Stepwise Self-Distillation for Object Detection](https://mlsanity.com/paper/2303.05015/) (2023)
- [A Comprehensive Study on Object Detection Techniques in Unconstrained Environments](https://mlsanity.com/paper/2304.05295/) (2023)
- [Image Recognition with Online Lightweight Vision Transformer: A Survey](https://mlsanity.com/paper/2505.03113/) (2025)
- [Task Integration Distillation for Object Detectors](https://mlsanity.com/paper/2404.01699/) (2024)
- [Architectural Insights into Knowledge Distillation for Object Detection: A Comprehensive Review](https://mlsanity.com/paper/2508.03317/) (2025)

## Abstract (arXiv)

Knowledge Distillation (KD) is a widely-used technology to inherit information from cumbersome teacher models to compact student models, consequently realizing model compression and acceleration. Compared with image classification, object detection is a more complex task, and designing specific KD methods for object detection is non-trivial. In this work, we elaborately study the behaviour difference between the teacher and student detection models, and obtain two intriguing observations: First, the teacher and student rank their detected candidate boxes quite differently, which results in their precision discrepancy. Second, there is a considerable gap between the feature response differences and prediction differences between teacher and student, indicating that equally imitating all the feature maps of the teacher is the sub-optimal choice for improving the student's accuracy. Based on the two observations, we propose Rank Mimicking (RM) and Prediction-guided Feature Imitation (PFI) for distilling one-stage detectors, respectively. RM takes the rank of candidate boxes from teachers as a new form of knowledge to distill, which consistently outperforms the traditional soft label distillation. PFI attempts to correlate feature differences with prediction differences, making feature imitation directly help to improve the student's accuracy. On MS COCO and PASCAL VOC benchmarks, extensive experiments are conducted on various detectors with different backbones to validate the effectiveness of our method. Specifically, RetinaNet with ResNet50 achieves 40.4% mAP in MS COCO, which is 3.5% higher than its baseline, and also outperforms previous KD methods.