mlsanityData to 7 Oct 2026

Focal and Global Knowledge Distillation for Detectors

arXiv 2111.11837 ↗First posted 23 Nov 2021cs.CVcited by 138

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation has been applied to image classification successfully. However, object detection is much more sophisticated and most knowledge distillation methods have failed on it. In this paper, we point out that in object detection, the features of the teacher and student vary greatly in different areas, especially in the foreground and background. If we distill them equally, the uneven differences between feature maps will negatively affect the distillation. Thus, we propose Focal and Global Distillation (FGD). Focal distillation separates the foreground and background, forcing the student to focus on the teacher's critical pixels and channels. Global distillation rebuilds the relation between different pixels and transfers it from teachers to students, compensating for missing global information in focal distillation. As our method only needs to calculate the loss on the feature map, FGD can be applied to various detectors. We experiment on various detectors with different backbones and the results show that the student detector achieves excellent mAP improvement. For example, ResNet-50 based RetinaNet, Faster RCNN, RepPoints and Mask RCNN with our distillation method achieve 40.7%, 42.0%, 42.0% and 42.1% mAP on COCO2017, which are 3.3, 3.6, 3.4 and 2.9 higher than the baseline, respectively. Our codes are available at https://github.com/yzd-v/FGD.

Key facts. Cited by 138 papers in the map (data through 7 Oct 2026). Builds on 34 of its 41 references that are in the map. First posted on arXiv 23 Nov 2021.

Builds on

34 of 41 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2017Mask R-CNNcited by 7,259
  8. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  9. 2017Squeeze-and-Excitation Networkscited by 4,763
  10. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  11. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  12. 2018YOLOv3: An Incremental Improvementcited by 2,652
22 more
  1. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  2. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
  3. 2017Non-local Neural Networkscited by 2,290
  4. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  5. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  6. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  7. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  8. 2021YOLOX: Exceeding YOLO Series in 2021cited by 1,064
  9. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  10. 2019CenterNet: Keypoint Triplets for Object Detectioncited by 654
  11. 2019Similarity-Preserving Knowledge Distillationcited by 507
  12. 2020Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detectioncited by 408
  13. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  14. 2019GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyondcited by 331
  15. 2017Relation Networks for Object Detectioncited by 305
  16. 2019SOLO: Segmenting Objects by Locationscited by 289
  17. 2019RepPoints: Point Set Representation for Object Detectioncited by 264
  18. 2019Distilling Object Detectors with Fine-grained Feature Imitationcited by 188
  19. 2021General Instance Distillation for Object Detectioncited by 113
  20. 2021Distilling Object Detectors via Decoupled Featurescited by 105
  21. 2021Instance-Conditional Knowledge Distillation for Object Detectioncited by 42
  22. 2020Distilling Object Detectors with Task Adaptive Regularizationcited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 22; 2023: 41; 2024: 36; 2025: 22; 2026: 16.

138 papers.

Sort:

By area:

  1. 2023Effective Whole-body Pose Estimation with Two-stages Distillationcited by 307
  2. 2022Masked Generative Distillationcited by 107
  3. 2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labelscited by 69
  4. 2022BEVDistill: Cross-Modal BEV Distillation for Multi-View 3D Object Detectioncited by 66
  5. 2023Object-Aware Distillation Pyramid for Open-Vocabulary Object Detectioncited by 63
  6. 2023Knowledge Diffusion for Distillationcited by 51
  7. 2023CrossKD: Cross-Head Knowledge Distillation for Object Detectioncited by 50
  8. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  9. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  10. 2022PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficientcited by 41
  11. 2023Adaptive Sparse Convolutional Networks with Global Context Enhancement for Faster Object Detection on Drone Imagescited by 41
  12. 2023DistillBEV: Boosting Multi-Camera 3D Object Detection with Cross-Modal Knowledge Distillationcited by 41
38 more
  1. 2024Tiny Machine Learning: Progress and Futurescited by 40
  2. 2022DETRDistill: A Universal Knowledge Distillation Framework for DETR-familiescited by 36
  3. 2022ViTKD: Practical Guidelines for ViT feature knowledge distillationcited by 34
  4. 2022MixSKD: Self-Knowledge Distillation from Mixup for Image Recognitioncited by 30
  5. 2022Towards Efficient 3D Object Detection with Knowledge Distillationcited by 29
  6. 2023X3KD: Knowledge Distillation Across Modalities, Tasks and Stages for Multi-Camera 3D Object Detectioncited by 28
  7. 2023FreeKD: Knowledge Distillation via Semantic Frequency Promptcited by 27
  8. 2024RadarDistill: Boosting Radar-based Object Detection Performance via Knowledge Distillation from LiDAR Featurescited by 27
  9. 2022Spatial Pruned Sparse Convolution for Efficient 3D Object Detectioncited by 26
  10. 2022Masked Distillation with Receptive Tokenscited by 23
  11. 2023NAYER: Noisy Layer Data Generation for Efficient and Effective Data-free Knowledge Distillationcited by 22
  12. 2023Function-Consistent Feature Distillationcited by 21
  13. 2022Exploring Inconsistent Knowledge Distillation for Object Detection with Data Augmentationcited by 19
  14. 2022PP-StructureV2: A Stronger Document Analysis Systemcited by 18
  15. 2023PanoSwin: a Pano-style Swin Transformer for Panorama Understandingcited by 18
  16. 2025ACAM-KD: Adaptive and Cooperative Attention Masking for Knowledge Distillationcited by 18
  17. 2022D3ETR: Decoder Distillation for Detection Transformercited by 17
  18. 2024Rethinking Centered Kernel Alignment in Knowledge Distillationcited by 17
  19. 2024On Efficient Variants of Segment Anything Model: A Surveycited by 16
  20. 2022Prediction-Guided Distillation for Dense Object Detectioncited by 14
  21. 2023Quantized Feature Distillation for Network Quantizationcited by 13
  22. 2022HEAD: HEtero-Assists Distillation for Heterogeneous Object Detectorscited by 12
  23. 2022Guided Hybrid Quantization for Object detection in Multimodal Remote Sensing Imagery via One-to-one Self-teachingcited by 12
  24. 2023DAMO-StreamNet: Optimizing Streaming Perception in Autonomous Drivingcited by 12
  25. 2023Gradient-Guided Knowledge Distillation for Object Detectorscited by 11
  26. 2023Cloud-Device Collaborative Learning for Multimodal Large Language Modelscited by 11
  27. 2024MedDet: Generative Adversarial Distillation for Efficient Cervical Disc Herniation Detectioncited by 11
  28. 2024ScaleKD: Strong Vision Transformers Could Be Excellent Teacherscited by 11
  29. 2023Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillationcited by 10
  30. 2024On the Road to Portability: Compressing End-to-End Motion Planner for Autonomous Drivingcited by 10
  31. 2022Rethinking Knowledge Distillation via Cross-Entropycited by 9
  32. 2023Spatial Self-Distillation for Object Detection with Inaccurate Bounding Boxescited by 9
  33. 2024LabelDistill: Label-guided Cross-modal Knowledge Distillation for Camera-based 3D Object Detectioncited by 9
  34. 2023Dual Relation Knowledge Distillation for Object Detectioncited by 8
  35. 2023Geometric-aware Pretraining for Vision-centric 3D Object Detectioncited by 8
  36. 2023Are Large Kernels Better Teachers than Transformers for ConvNets?cited by 8
  37. 2023Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentationcited by 8
  38. 2024Active Object Detection with Knowledge Aggregation and Distillation from Large Modelscited by 8

50 most cited of 138 All 138 (JSON)