mlsanityData to 14 Oct 2026

Instance-Conditional Knowledge Distillation for Object Detection

arXiv 2110.12724 ↗First posted 25 Oct 2021cs.CVcited by 42

On the map: Knowledge distillation → Detection and dense prediction distillation

Abstract (arXiv)

Knowledge distillation has shown great success in classification, however, it is still challenging for detection. In a typical image for detection, representations from different locations may have different contributions to detection targets, making the distillation hard to balance. In this paper, we propose a conditional distillation framework to distill the desired knowledge, namely knowledge that is beneficial in terms of both classification and localization for every instance. The framework introduces a learnable conditional decoding module, which retrieves information given each target instance as query. Specifically, we encode the condition information as query and use the teacher's representations as key. The attention between query and key is used to measure the contribution of different features, guided by a localization-recognition-sensitive auxiliary task. Extensive experiments demonstrate the efficacy of our method: we observe impressive improvements under various settings. Notably, we boost RetinaNet with ResNet-50 backbone from 37.4 to 40.7 mAP (+3.3) under 1x schedule, that even surpasses the teacher (40.4 mAP) with ResNet-101 backbone under 3x schedule. Code has been released on https://github.com/megvii-research/ICD.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation. Builds on 37 of its 50 references that are in the map. First posted on arXiv 25 Oct 2021.

Builds on

37 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Distilling Object Detectors via Decoupled Features26 Mar
  2. 2021General Instance Distillation for Object Detection3 Mar

Other references · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2017Mask R-CNNcited by 7,277
  9. 2016Densely Connected Convolutional Networkscited by 6,720
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  11. 2016Layer Normalizationcited by 5,669
  12. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
23 more
  1. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  2. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  3. 2015VQA: Visual Question Answeringcited by 2,506
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  5. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  6. 2020Knowledge Distillation: A Surveycited by 1,603
  7. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,577
  8. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  10. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,276
  11. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  12. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  13. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  14. 2018CornerNet: Detecting Objects as Paired Keypointscited by 763
  15. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  16. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  17. 2018Stacked Cross Attention for Image-Text Matchingcited by 416
  18. 2020Conditional Convolutions for Instance Segmentationcited by 314
  19. 2017Relation Networks for Object Detectioncited by 305
  20. 2019Distilling Object Detectors with Fine-grained Feature Imitationcited by 189
  21. 2020End-to-End Object Detection with Fully Convolutional Networkcited by 70
  22. 2020Explaining Knowledge Distillation by Quantifying the Knowledgecited by 66
  23. 2021Points as Queries: Weakly Semi-supervised Object Detection by Pointscited by 60

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 15; 2023: 14; 2024: 9; 2025: 2; 2026: 1.

42 papers.

Sort:

By area:

  1. 2021Focal and Global Knowledge Distillation for Detectorscited by 142
  2. 2022Masked Generative Distillationcited by 107
  3. 2023Object-Aware Distillation Pyramid for Open-Vocabulary Object Detectioncited by 63
  4. 2022PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficientcited by 41
  5. 2023DistillBEV: Boosting Multi-Camera 3D Object Detection with Cross-Modal Knowledge Distillationcited by 41
  6. 2022PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D Detectioncited by 33
  7. 2024RadarDistill: Boosting Radar-based Object Detection Performance via Knowledge Distillation from LiDAR Featurescited by 28
  8. 2023Function-Consistent Feature Distillationcited by 21
  9. 2022Localization Distillation for Object Detectioncited by 19
  10. 2024Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillationcited by 19
  11. 2022D3ETR: Decoder Distillation for Detection Transformercited by 17
  12. 2022Object Detection in Autonomous Vehicles: Status and Open Challengescited by 14
30 more
  1. 2022Prediction-Guided Distillation for Dense Object Detectioncited by 14
  2. 2022Structural Knowledge Distillation for Object Detectioncited by 14
  3. 2022KD-DETR: Knowledge Distillation for Detection Transformer with Consistent Distillation Points Samplingcited by 13
  4. 2023Gradient-Guided Knowledge Distillation for Object Detectorscited by 12
  5. 2024ScaleKD: Strong Vision Transformers Could Be Excellent Teacherscited by 11
  6. 2023Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillationcited by 10
  7. 2024On the Road to Portability: Compressing End-to-End Motion Planner for Autonomous Drivingcited by 10
  8. 2023Geometric-aware Pretraining for Vision-centric 3D Object Detectioncited by 8
  9. 2024Active Object Detection with Knowledge Aggregation and Distillation from Large Modelscited by 8
  10. 2023DOT: A Distillation-Oriented Trainercited by 7
  11. 2023Let All be Whitened: Multi-teacher Distillation for Efficient Visual Retrievalcited by 7
  12. 2022Boosting Single-Frame 3D Object Detection by Simulating Multi-Frame Point Cloudscited by 6
  13. 2022Label-Guided Auxiliary Training Improves 3D Object Detectorcited by 6
  14. 2023Bridging Cross-task Protocol Inconsistency for Distillation in Dense Object Detectioncited by 6
  15. 2023Object-centric Cross-modal Feature Distillation for Event-based Object Detectioncited by 6
  16. 2022Cross-domain Federated Object Detectioncited by 5
  17. 2022Task-Balanced Distillation for Object Detectioncited by 5
  18. 2024Knowledge Distillation via Query Selection for Detection Transformercited by 5
  19. 2022BEV-LGKD: A Unified LiDAR-Guided Knowledge Distillation Framework for BEV 3D Object Detectioncited by 4
  20. 2025Distilling 3D distinctive local descriptors for 6D pose estimationcited by 4
  21. 2023Knowledge Distillation for Object Detection: from generic to remote sensing datasetscited by 3
  22. 2023RdimKD: Generic Distillation Paradigm by Dimensionality Reductioncited by 3
  23. 2023Smooth and Stepwise Self-Distillation for Object Detectioncited by 2
  24. 2024GraphKD: Exploring Knowledge Distillation Towards Document Object Detection with Structured Graph Creationcited by 2
  25. 2024Teaching with Uncertainty: Unleashing the Potential of Knowledge Distillation in Object Detectioncited by 1
  26. 2025CanKD: Cross-Attention-based Non-local operation for Feature-based Knowledge Distillationcited by 1
  27. 2022Heterogeneous Feature Distillation Network for SAR Image Semantic Segmentationcited by 0
  28. 2023Efficient Visual Fault Detection for Freight Train Braking System via Heterogeneous Self Distillation in the Wildcited by 0
  29. 2024Generative Denoise Distillation: Simple Stochastic Noises Induce Efficient Knowledge Transfer for Dense Predictioncited by 0
  30. 2026FD-CanKD: Frequency-Decoupled Cross-Attention Distillation as a Refinement Prior for Compact Object Detectorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Distilling Image Classifiers in Object Detectors
  2. 2021Localization Distillation for Dense Object Detection
  3. 2025Architectural Insights into Knowledge Distillation for Object Detection: A Comprehensive Review
  4. 2021Revisiting Knowledge Distillation for Object Detection
  5. 2024Task Integration Distillation for Object Detectors
  6. 2023SKDF: A Simple Knowledge Distillation Framework for Distilling Open-Vocabulary Knowledge to Open-world Object Detector
  7. 2023DMKD: Improving Feature-based Knowledge Distillation for Object Detection Via Dual Masking Augmentation
  8. 2021Deep Structured Instance Graph for Distilling Object Detectors