mlsanityData to 7 Oct 2026

Masked Generative Distillation

arXiv 2205.01529 ↗First posted 3 May 2022cs.CVcited by 107

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation has been applied to various tasks successfully. The current distillation algorithm usually improves students' performance by imitating the output of the teacher. This paper shows that teachers can also improve students' representation power by guiding students' feature recovery. From this point of view, we propose Masked Generative Distillation (MGD), which is simple: we mask random pixels of the student's feature and force it to generate the teacher's full feature through a simple block. MGD is a truly general feature-based distillation method, which can be utilized on various tasks, including image classification, object detection, semantic segmentation and instance segmentation. We experiment on different models with extensive datasets and the results show that all the students achieve excellent improvements. Notably, we boost ResNet-18 from 69.90% to 71.69% ImageNet top-1 accuracy, RetinaNet with ResNet-50 backbone from 37.4 to 41.0 Boundingbox mAP, SOLO based on ResNet-50 from 33.1 to 36.2 Mask mAP and DeepLabV3 based on ResNet-18 from 73.20 to 76.02 mIoU. Our codes are available at https://github.com/yzd-v/MGD.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Builds on 33 of its 39 references that are in the map. First posted on arXiv 3 May 2022.

Builds on

33 of 39 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2017Mask R-CNNcited by 7,259
  7. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  8. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  9. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  10. 2022A ConvNet for the 2020scited by 3,545
  11. 2016Pyramid Scene Parsing Networkcited by 2,472
  12. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
21 more
  1. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  2. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  3. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  4. 2019Relational Knowledge Distillationcited by 873
  5. 2019Contrastive Representation Distillationcited by 734
  6. 2019Res2Net: A New Multi-scale Backbone Architecturecited by 468
  7. 2019On the Efficacy of Knowledge Distillationcited by 377
  8. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  9. 2019SOLO: Segmenting Objects by Locationscited by 289
  10. 2021Distilling Knowledge via Knowledge Reviewcited by 285
  11. 2019Structured Knowledge Distillation for Dense Predictioncited by 277
  12. 2019RepPoints: Point Set Representation for Object Detectioncited by 264
  13. 2019Distilling Object Detectors with Fine-grained Feature Imitationcited by 188
  14. 2020Channel-wise Knowledge Distillation for Dense Predictioncited by 161
  15. 2021Focal and Global Knowledge Distillation for Detectorscited by 138
  16. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  17. 2021General Instance Distillation for Object Detectioncited by 113
  18. 2019Knowledge Adaptation for Efficient Semantic Segmentationcited by 105
  19. 2021Distilling Object Detectors via Decoupled Featurescited by 105
  20. 2021Instance-Conditional Knowledge Distillation for Object Detectioncited by 42
  21. 2021Distilling Object Detectors with Feature Richnesscited by 38

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 34; 2024: 27; 2025: 17; 2026: 16.

107 papers.

Sort:

By area:

  1. 2023Effective Whole-body Pose Estimation with Two-stages Distillationcited by 307
  2. 2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labelscited by 69
  3. 2022BEVDistill: Cross-Modal BEV Distillation for Multi-View 3D Object Detectioncited by 66
  4. 2023Object-Aware Distillation Pyramid for Open-Vocabulary Object Detectioncited by 63
  5. 2023UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformerscited by 51
  6. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  7. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  8. 2024Not All Voxels Are Equal: Hardness-Aware Semantic Scene Completion with Self-Distillationcited by 43
  9. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  10. 2022DAMO-YOLO : A Report on Real-Time Object Detection Designcited by 38
  11. 2022DETRDistill: A Universal Knowledge Distillation Framework for DETR-familiescited by 36
  12. 2022ViTKD: Practical Guidelines for ViT feature knowledge distillationcited by 34
38 more
  1. 2024Masked Image Modeling: A Surveycited by 28
  2. 2023FreeKD: Knowledge Distillation via Semantic Frequency Promptcited by 27
  3. 2022Exploring Inconsistent Knowledge Distillation for Object Detection with Data Augmentationcited by 19
  4. 2023LiDAR2Map: In Defense of LiDAR-Based Semantic Map Construction Using Online Camera Distillationcited by 18
  5. 2024Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillationcited by 18
  6. 2025ACAM-KD: Adaptive and Cooperative Attention Masking for Knowledge Distillationcited by 18
  7. 2022D3ETR: Decoder Distillation for Detection Transformercited by 17
  8. 2024Rethinking Centered Kernel Alignment in Knowledge Distillationcited by 17
  9. 2023CM-MaskSD: Cross-Modality Masked Self-Distillation for Referring Image Segmentationcited by 15
  10. 2023BPKD: Boundary Privileged Knowledge Distillation For Semantic Segmentationcited by 15
  11. 2024AMFD: Distillation via Adaptive Multimodal Fusion for Multispectral Pedestrian Detectioncited by 15
  12. 2022KD-DETR: Knowledge Distillation for Detection Transformer with Consistent Distillation Points Samplingcited by 12
  13. 2023NormKD: Normalized Logits for Knowledge Distillationcited by 12
  14. 2023Student-friendly Knowledge Distillationcited by 11
  15. 2023VanillaKD: Revisit the Power of Vanilla Knowledge Distillation from Small Scale to Large Scalecited by 11
  16. 2024Towards Robust and Efficient Cloud-Edge Elastic Model Adaptation via Selective Entropy Distillationcited by 11
  17. 2024MedDet: Generative Adversarial Distillation for Efficient Cervical Disc Herniation Detectioncited by 11
  18. 2024ScaleKD: Strong Vision Transformers Could Be Excellent Teacherscited by 11
  19. 2023Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillationcited by 10
  20. 2022Rethinking Knowledge Distillation via Cross-Entropycited by 9
  21. 2024Label-efficient Semantic Scene Completion with Scribble Annotationscited by 9
  22. 2022Towards Sustainable Self-supervised Learningcited by 8
  23. 2023Are Large Kernels Better Teachers than Transformers for ConvNets?cited by 8
  24. 2023Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentationcited by 8
  25. 2023Avatar Knowledge Distillation: Self-ensemble Teacher Paradigm with Uncertaintycited by 7
  26. 2024Direct Distillation between Different Domainscited by 7
  27. 2023Bridging Cross-task Protocol Inconsistency for Distillation in Dense Object Detectioncited by 6
  28. 2023Object-centric Cross-modal Feature Distillation for Event-based Object Detectioncited by 6
  29. 2024MergeNet: Knowledge Migration across Heterogeneous Models, Tasks, and Modalitiescited by 6
  30. 2024Decoupling Dark Knowledge via Block-wise Logit Distillation for Feature-level Alignmentcited by 6
  31. 2025CLoCKDistill: Consistent Location-and-Context-aware Knowledge Distillation for DETRscited by 6
  32. 2022Task-Balanced Distillation for Object Detectioncited by 5
  33. 2023A Simple and Generic Framework for Feature Distillation via Channel-wise Transformationcited by 5
  34. 2023Towards Efficient Task-Driven Model Reprogramming with Foundation Modelscited by 5
  35. 2023PEA-Diffusion: Parameter-Efficient Adapter with Knowledge Distillation in non-English Text-to-Image Generationcited by 5
  36. 2024Feature Denoising Diffusion Model for Blind Image Quality Assessmentcited by 5
  37. 2024Aligning in a Compact Space: Contrastive Knowledge Distillation between Heterogeneous Architecturescited by 5
  38. 2024Reprogramming Distillation for Medical Foundation Modelscited by 5

50 most cited of 107 All 107 (JSON)