# G-DetKD: Towards General Distillation Framework for Object Detectors via Contrastive and Semantic-guided Feature Imitation

arXiv: https://arxiv.org/abs/2108.07482 · HTML: https://mlsanity.com/paper/2108.07482/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 31 of its 41 references that are in the map.
- First posted on arXiv 17 Aug 2021.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 30 of 31)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection](https://mlsanity.com/paper/1912.02424/) (2019)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection](https://mlsanity.com/paper/1904.07392/) (2019)
- [RepPoints: Point Set Representation for Object Detection](https://mlsanity.com/paper/1904.11490/) (2019)
- [Distilling Object Detectors with Fine-grained Feature Imitation](https://mlsanity.com/paper/1906.03609/) (2019)
- [Probabilistic Anchor Assignment with IoU Prediction for Object Detection](https://mlsanity.com/paper/2007.08103/) (2020)
- [InstaBoost: Boosting Instance Segmentation via Probability Map Guided Copy-Pasting](https://mlsanity.com/paper/1908.07801/) (2019)
- [Soft Anchor-Point Object Detection](https://mlsanity.com/paper/1911.12448/) (2019)
- [BorderDet: Border Feature for Dense Object Detection](https://mlsanity.com/paper/2007.11056/) (2020)
- [Distilling Object Detectors with Task Adaptive Regularization](https://mlsanity.com/paper/2006.13108/) (2020)
- [SM-NAS: Structural-to-Modular Neural Architecture Search for Object Detection](https://mlsanity.com/paper/1911.09929/) (2019)

## Built on by (most cited first, 28 of 28)

- [DetGPT: Detect What You Need via Reasoning](https://mlsanity.com/paper/2305.14167/) (2023)
- [CLIP-KD: An Empirical Study of CLIP Model Distillation](https://mlsanity.com/paper/2307.12732/) (2023)
- [Object-Aware Distillation Pyramid for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2303.05892/) (2023)
- [PerceptionGPT: Effectively Fusing Visual Perception into LLM](https://mlsanity.com/paper/2311.06612/) (2023)
- [Contrastive Mean Teacher for Domain Adaptive Object Detectors](https://mlsanity.com/paper/2305.03034/) (2023)
- [CrossKD: Cross-Head Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2306.11369/) (2023)
- [Model Agnostic Sample Reweighting for Out-of-Distribution Learning](https://mlsanity.com/paper/2301.09819/) (2023)
- [Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptions](https://mlsanity.com/paper/2406.07502/) (2024)
- [PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient](https://mlsanity.com/paper/2207.02039/) (2022)
- [Probabilistic Bilevel Coreset Selection](https://mlsanity.com/paper/2301.09880/) (2023)
- [ConQueR: Query Contrast Voxel-DETR for 3D Object Detection](https://mlsanity.com/paper/2212.07289/) (2022)
- [LGD: Label-guided Self-distillation for Object Detection](https://mlsanity.com/paper/2109.11496/) (2021)
- [Prediction-Guided Distillation for Dense Object Detection](https://mlsanity.com/paper/2203.05469/) (2022)
- [DAMO-StreamNet: Optimizing Streaming Perception in Autonomous Driving](https://mlsanity.com/paper/2303.17144/) (2023)
- [HEAD: HEtero-Assists Distillation for Heterogeneous Object Detectors](https://mlsanity.com/paper/2207.05345/) (2022)
- [Modeling Continuous Motion for 3D Point Cloud Object Tracking](https://mlsanity.com/paper/2303.07605/) (2023)
- [Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentation](https://mlsanity.com/paper/2312.04168/) (2023)
- [Teach-DETR: Better Training DETR with Teachers](https://mlsanity.com/paper/2211.11953/) (2022)
- [Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2504.13825/) (2025)
- [U-Know-DiffPAN: An Uncertainty-aware Knowledge Distillation Diffusion Framework with Details Enhancement for PAN-Sharpening](https://mlsanity.com/paper/2412.06243/) (2024)
- [VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training](https://mlsanity.com/paper/2506.13888/) (2025)
- [DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection](https://mlsanity.com/paper/2604.02753/) (2026)
- [Smooth and Stepwise Self-Distillation for Object Detection](https://mlsanity.com/paper/2303.05015/) (2023)
- [Towards Better Modeling with Missing Data: A Contrastive Learning-based Visual Analytics Perspective](https://mlsanity.com/paper/2309.09744/) (2023)
- [Architectural Insights into Knowledge Distillation for Object Detection: A Comprehensive Review](https://mlsanity.com/paper/2508.03317/) (2025)
- [Contrast-Guided Cross-Modal Distillation for Thermal Object Detection](https://mlsanity.com/paper/2511.01435/) (2025)
- [DFMSD: Dual Feature Masking Stage-wise Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2407.13147/) (2024)
- [Training a Student Expert via Semi-Supervised Foundation Model Distillation](https://mlsanity.com/paper/2604.03841/) (2026)

## Abstract (arXiv)

In this paper, we investigate the knowledge distillation (KD) strategy for object detection and propose an effective framework applicable to both homogeneous and heterogeneous student-teacher pairs. The conventional feature imitation paradigm introduces imitation masks to focus on informative foreground areas while excluding the background noises. However, we find that those methods fail to fully utilize the semantic information in all feature pyramid levels, which leads to inefficiency for knowledge distillation between FPN-based detectors. To this end, we propose a novel semantic-guided feature imitation technique, which automatically performs soft matching between feature pairs across all pyramid levels to provide the optimal guidance to the student. To push the envelop even further, we introduce contrastive distillation to effectively capture the information encoded in the relationship between different feature regions. Finally, we propose a generalized detection KD pipeline, which is capable of distilling both homogeneous and heterogeneous detector pairs. Our method consistently outperforms the existing detection KD techniques, and works when (1) components in the framework are used separately and in conjunction; (2) for both homogeneous and heterogenous student-teacher pairs and (3) on multiple detection benchmarks. With a powerful X101-FasterRCNN-Instaboost detector as the teacher, R50-FasterRCNN reaches 44.0% AP, R50-RetinaNet reaches 43.3% AP and R50-FCOS reaches 43.1% AP on COCO dataset.