mlsanityData to 14 Oct 2026

CrossKD: Cross-Head Knowledge Distillation for Object Detection

arXiv 2306.11369 ↗First posted 20 Jun 2023cs.CVcited by 52

On the map: Knowledge distillation → Detection and dense prediction distillation

Abstract (arXiv)

Knowledge Distillation (KD) has been validated as an effective model compression technique for learning compact object detectors. Existing state-of-the-art KD methods for object detection are mostly based on feature imitation. In this paper, we present a general and effective prediction mimicking distillation scheme, called CrossKD, which delivers the intermediate features of the student's detection head to the teacher's detection head. The resulting cross-head predictions are then forced to mimic the teacher's predictions. This manner relieves the student's head from receiving contradictory supervision signals from the annotations and the teacher's predictions, greatly improving the student's detection performance. Moreover, as mimicking the teacher's predictions is the target of KD, CrossKD offers more task-oriented information in contrast with feature imitation. On MS COCO, with only prediction mimicking losses applied, our CrossKD boosts the average precision of GFL ResNet-50 with 1x training schedule from 40.2 to 43.7, outperforming all existing KD methods. In addition, our method also works well when distilling detectors with heterogeneous backbones. Code is available at https://github.com/jbwang1997/CrossKD.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation. Builds on 58 of its 78 references that are in the map. First posted on arXiv 20 Jun 2023.

Builds on

58 of 78 references.

Closest ancestors · 10

Same direction, newest first

  1. 2022Structural Knowledge Distillation for Object Detection23 Nov
  2. 2022HEAD: HEtero-Assists Distillation for Heterogeneous Object Detectors12 Jul
  3. 2022PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient5 Jul
  4. 2021Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-guided Feature Imitation9 Dec
  5. 2021Focal and Global Knowledge Distillation for Detectors23 Nov
  6. 2021G-DetKD: Towards General Distillation Framework for Object Detectors via Contrastive and Semantic-guided Feature Imitation17 Aug
  7. 2021Online Knowledge Distillation for Efficient Pose Estimation4 Aug
  8. 2021Distilling Object Detectors via Decoupled Features26 Mar
  9. 2021General Instance Distillation for Object Detection3 Mar
  10. 2021Localization Distillation for Dense Object Detection24 Feb

Other references · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2017Mask R-CNNcited by 7,277
  8. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  9. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  10. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  11. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  12. 2015Fast R-CNNcited by 2,933
36 more
  1. 2018YOLOv3: An Incremental Improvementcited by 2,659
  2. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  3. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  4. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,630
  5. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,577
  6. 2016YOLO9000: Better, Faster, Strongercited by 1,431
  7. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  8. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,368
  9. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  10. 2021YOLOX: Exceeding YOLO Series in 2021cited by 1,071
  11. 2019Relational Knowledge Distillationcited by 889
  12. 2017Deep Mutual Learningcited by 628
  13. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  14. 2022DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRcited by 573
  15. 2019Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selectioncited by 540
  16. 2022DN-DETR: Accelerate DETR Training by Introducing Query DeNoisingcited by 488
  17. 2019Hybrid Task Cascade for Instance Segmentationcited by 481
  18. 2021Conditional DETR for Fast Training Convergencecited by 424
  19. 2020Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detectioncited by 409
  20. 2022Decoupled Knowledge Distillationcited by 388
  21. 2019On the Efficacy of Knowledge Distillationcited by 381
  22. 2019A Comprehensive Overhaul of Feature Distillationcited by 348
  23. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 260
  24. 2021TOOD: Task-aligned One-stage Object Detectioncited by 218
  25. 2022RTMDet: An Empirical Study of Designing Real-Time Object Detectorscited by 195
  26. 2019Distilling Object Detectors with Fine-grained Feature Imitationcited by 189
  27. 2019Feature Selective Anchor-Free Module for Single-Shot Object Detectioncited by 139
  28. 2022Curriculum Temperature for Knowledge Distillationcited by 112
  29. 2020Generalized Focal Loss V2: Learning Reliable Localization Quality Estimation for Dense Object Detectioncited by 104
  30. 2018Snapshot Distillation: Teacher-Student Optimization in One Generationcited by 99
  31. 2019Region Proposal by Guided Anchoringcited by 87
  32. 2021Distilling Object Detectors with Feature Richnesscited by 38
  33. 2019Few Shot Network Compression via Cross Distillationcited by 26
  34. 2023Function-Consistent Feature Distillationcited by 21
  35. 2021Improving Object Detection by Label Assignment Distillationcited by 17
  36. 2022ARM3D: Attention-based relation module for indoor 3D object detectioncited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 26; 2026: 15.

52 papers.

Sort:

By area:

  1. 2025ACAM-KD: Adaptive and Cooperative Attention Masking for Knowledge Distillationcited by 18
  2. 2024ScaleKD: Strong Vision Transformers Could Be Excellent Teacherscited by 11
  3. 2025GenRecal: Generation after Recalibration from Large to Small Vision-Language Modelscited by 10
  4. 2024ComKD-CLIP: Comprehensive Knowledge Distillation for Contrastive Language-Image Pre-traning Modelcited by 9
  5. 2024MiniVLN: Efficient Vision-and-Language Navigation by Progressive Knowledge Distillationcited by 6
  6. 2025Generalized Diffusion Detector: Mining Robust Features from Diffusion Models for Domain-Generalized Detectioncited by 6
  7. 2025Masking Teacher and Reinforcing Student for Distilling Vision-Language Modelscited by 6
  8. 2024MutDet: Mutually Optimizing Pre-training for Remote Sensing Object Detectioncited by 5
  9. 2025Uncertainty-Aware Knowledge Distillation for Compact and Efficient 6DoF Pose Estimationcited by 5
  10. 2025BlurDM: A Blur Diffusion Model for Image Deblurringcited by 5
  11. 2024Continual Panoptic Perception: Towards Multi-modal Incremental Interpretation of Remote Sensing Imagescited by 4
  12. 2024Domain-invariant Progressive Knowledge Distillation for UAV-based Object Detectioncited by 4
38 more
  1. 2025Unimodal-driven Distillation in Multimodal Emotion Recognition with Dynamic Fusioncited by 4
  2. 2025Sample-level Adaptive Knowledge Distillation for Action Recognitioncited by 4
  3. 2025Knowledge Distillation for Image Restoration : Simultaneous Learning from Degraded and Clean Imagescited by 3
  4. 2023AICSD: Adaptive Inter-Class Similarity Distillation for Semantic Segmentationcited by 2
  5. 2025CAE-DFKD: Bridging the Transferability Gap in Data-Free Knowledge Distillationcited by 2
  6. 2025Image Recognition with Online Lightweight Vision Transformer: A Surveycited by 2
  7. 2025Boosting Domain Generalized and Adaptive Detection with Diffusion Models: Fitness, Generalization, and Transferabilitycited by 2
  8. 2025BridgeTA: Bridging the Representation Gap in Knowledge Distillation via Teacher Assistant for Bird's Eye View Map Segmentationcited by 2
  9. 2024Graph Relation Distillation for Efficient Biomedical Instance Segmentationcited by 1
  10. 2024Wisdom of Committee: Diverse Distillation from Large Foundation Models and Domain Expertscited by 1
  11. 2025Teaching in adverse scenes: a statistically feedback-driven threshold and mask adjustment teacher-student framework for object detection in UAV images under adverse scenescited by 1
  12. 2025Uncertainty-Aware Cross-Modal Knowledge Distillation with Prototype Learning for Multimodal Brain-Computer Interfacescited by 1
  13. 2025Architectural Insights into Knowledge Distillation for Object Detection: A Comprehensive Reviewcited by 1
  14. 2025Designing Object Detection Models for TinyML: Foundations, Comparative Analysis, Challenges, and Emerging Solutionscited by 1
  15. 2025Dual-Stream Spectral Decoupling Distillation for Remote Sensing Object Detectioncited by 1
  16. 2026HNDiff: Haze-Noise Diffusion for Image Dehazingcited by 1
  17. 2024Towards Customized Knowledge Distillation for Chip-Level Dense Image Predictionscited by 0
  18. 2024Cross-Domain Knowledge Distillation for Low-Resolution Human Pose Estimationcited by 0
  19. 2025Feature Representation Transferring to Lightweight Models via Perception Coherencecited by 0
  20. 2025MoKD: Multi-Task Optimization for Knowledge Distillationcited by 0
  21. 2025Knowledge Distillation for Reservoir-based Classifier: Human Activity Recognitioncited by 0
  22. 2025Feature Fusion and Knowledge-Distilled Multi-Modal Multi-Target Detectioncited by 0
  23. 2025Distilling Balanced Knowledge from a Biased Teachercited by 0
  24. 2025HKT: A Biologically Inspired Framework for Modular Hereditary Knowledge Transfer in Neural Networkscited by 0
  25. 2025Online Data Curation for Object Detection via Marginal Contributions to Dataset-level Average Precisioncited by 0
  26. 2025Uncertainty-Resilient Multimodal Learning via Consistency-Guided Cross-Modal Transfercited by 0
  27. 2026Evolving Without Ending: Unifying Multimodal Incremental Learning for Continual Panoptic Perceptioncited by 0
  28. 2026CD-FKD: Cross-Domain Feature Knowledge Distillation for Robust Single-Domain Generalization in Object Detectioncited by 0
  29. 2026Bridge: Basis-Driven Causal Inference Marries VFMs for Domain Generalizationcited by 0
  30. 2026Edge AI for Automotive Vulnerable Road User Safety: Deployable Detection via Knowledge Distillationcited by 0
  31. 2026AnyDepth-DETR/-YOLO: Any-depth object detection with a single networkcited by 0
  32. 2026Denoising-Enhanced Coarse-to-Fine Infrared Small Target Detection with Attention Prior-Guided Knowledge Distillationcited by 0
  33. 2026Heterogeneous Knowledge Distillation via Geometry Decoupling and Momentum-Aware Gradient Regulationcited by 0
  34. 2026Channel-wise Dynamic Knowledge Distillation via Adaptive Sample Generation for Action Recognitioncited by 0
  35. 2026EGM-Det: Entropy-Guided Multimodal Adaptive Fusion for UAV RGB-IR Object Detectioncited by 0
  36. 2026Socialized Detector Learning: Trajectory-Guided and Reciprocal Distillation for Heterogeneous Object Detectorscited by 0
  37. 2026Knowledge Distillation Driven Semantic NOMA with GAN Refinement for 6G Robotic Vehicle Networkscited by 0
  38. 2026DTKDP: A Dual Teacher Knowledge Distillation and Pruning Framework for Lightweight Oriented SAR Ship Detectioncited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Gradient-Guided Knowledge Distillation for Object Detectors
  2. 2021Beyond Classification: Knowledge Distillation using Multi-Object Impressions
  3. 2026FD-CanKD: Frequency-Decoupled Cross-Attention Distillation as a Refinement Prior for Compact Object Detectors
  4. 2022Prediction-Guided Distillation for Dense Object Detection
  5. 2022Cross Domain Object Detection by Target-Perceived Dual Branch Distillation
  6. 2022Localization Distillation for Object Detection
  7. 2021Distilling Image Classifiers in Object Detectors
  8. 2023Bridging Cross-task Protocol Inconsistency for Distillation in Dense Object Detection