mlsanityData to 14 Oct 2026

MixSKD: Self-Knowledge Distillation from Mixup for Image Recognition

arXiv 2208.05768 ↗First posted 11 Aug 2022cs.CVcited by 30

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Unlike the conventional Knowledge Distillation (KD), Self-KD allows a network to learn knowledge from itself without any guidance from extra networks. This paper proposes to perform Self-KD from image Mixture (MixSKD), which integrates these two techniques into a unified framework. MixSKD mutually distills feature maps and probability distributions between the random pair of original images and their mixup images in a meaningful way. Therefore, it guides the network to learn cross-image knowledge by modelling supervisory signals from mixup images. Moreover, we construct a self-teacher network by aggregating multi-stage feature maps for providing soft labels to supervise the backbone classifier, further improving the efficacy of self-boosting. Experiments on image classification and transfer learning to object detection and semantic segmentation demonstrate that MixSKD outperforms other state-of-the-art Self-KD and data augmentation methods. The code is available at https://github.com/winycg/Self-KD-Lib.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 40 of its 59 references that are in the map. First posted on arXiv 11 Aug 2022.

Builds on

40 of 59 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Knowledge Distillation Using Hierarchical Self-Supervision Augmented Distribution7 Sep
  2. 2021Hierarchical Self-supervised Augmented Knowledge Distillation29 Jul

Other references · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  6. 2016Densely Connected Convolutional Networkscited by 6,720
  7. 2022Generative Adversarial Networkscited by 5,627
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  10. 2016Wide Residual Networkscited by 3,070
  11. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  12. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
26 more
  1. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  2. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  3. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  4. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  5. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  6. 2017Random Erasing Data Augmentationcited by 1,239
  7. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 920
  8. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  9. 2017Deep Mutual Learningcited by 628
  10. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  11. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 429
  12. 2019Correlation Congruence for Knowledge Distillationcited by 285
  13. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  14. 2019Learning Lightweight Lane Detection CNNs by Self Attention Distillationcited by 162
  15. 2020Regularizing Class-wise Predictions via Self-knowledge Distillationcited by 144
  16. 2021Focal and Global Knowledge Distillation for Detectorscited by 142
  17. 2019Online Knowledge Distillation with Diverse Peerscited by 128
  18. 2022Cross-Image Relational Knowledge Distillation for Semantic Segmentationcited by 118
  19. 2021Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillationcited by 46
  20. 2021Mutual Contrastive Learning for Visual Representation Learningcited by 42
  21. 2021Multi-View Correlation Distillation for Incremental Object Detectioncited by 36
  22. 2020Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation from a Blackbox Modelcited by 28
  23. 2019Deeply-supervised Knowledge Synergycited by 18
  24. 2019Gated Convolutional Networks with Hybrid Connectivity for Image Classificationcited by 14
  25. 2020Multi-view Contrastive Learning for Online Knowledge Distillationcited by 11
  26. 2022CoupleFace: Relation Matters for Face Recognition Distillationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 9; 2025: 5; 2026: 5.

30 papers.

Sort:

By area:

  1. 2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 87
  2. 2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labelscited by 69
  3. 2024Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitatecited by 47
  4. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 20
  5. 2024A Survey on Mixup Augmentations and Beyondcited by 20
  6. 2025Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognitioncited by 16
  7. 2023Jaccard Metric Losses: Optimizing the Jaccard Index with Soft Labelscited by 12
  8. 2023Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcementcited by 11
  9. 2024A Comprehensive Review of Knowledge Distillation in Computer Visioncited by 10
  10. 2023Learning Contrastive Self-Distillation for Ultra-Fine-Grained Visual Categorization Targeting Limited Samplescited by 9
  11. 2024Self-Adapting Large Visual-Language Models to Edge Devices across Visual Modalitiescited by 7
  12. 2025Test-Time Distillation for Continual Model Adaptationcited by 5
18 more
  1. 2023Masking meets Supervision: A Strong Learning Alliancecited by 3
  2. 2023RdimKD: Generic Distillation Paradigm by Dimensionality Reductioncited by 3
  3. 2023Distilling Calibrated Student from an Uncalibrated Teachercited by 2
  4. 2023Infinite Class Mixupcited by 2
  5. 2024Online Policy Distillation with Decision-Attentioncited by 2
  6. 2024Multi-Branch Mutual-Distillation Transformer for EEG-Based Seizure Subtype Classificationcited by 2
  7. 2024SalNAS: Efficient Saliency-prediction Neural Architecture Search with self-knowledge distillationcited by 1
  8. 2024Prototype-Driven Multi-Feature Generation for Visible-Infrared Person Re-identificationcited by 1
  9. 2025LeAD-M3D: Leveraging Asymmetric Distillation for Real-Time Monocular 3D Detectioncited by 1
  10. 2026Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictionscited by 1
  11. 2023Team AcieLee: Technical Report for EPIC-SOUNDS Audio-Based Interaction Recognition Challenge 2023cited by 0
  12. 2024LNPT: Label-free Network Pruning and Trainingcited by 0
  13. 2025Intra-class Patch Swap for Self-Distillationcited by 0
  14. 2025SST: Self-training with Self-adaptive Thresholding for Semi-supervised Learningcited by 0
  15. 2026Teacher-Guided Student Self-Knowledge Distillation Using Diffusion Modelcited by 0
  16. 2026D3O: Dynamic Distribution Distillation for Ordinal Regressioncited by 0
  17. 2026Cross-Domain, Multi-Task Data-to-Text Generation without In-Domain Training Datacited by 0
  18. 2026Efficient Quantization-Aware Distillation with Cross-Modal Alignment for Edge Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Understanding the Role of Mixup in Knowledge Distillation: An Empirical Study
  2. 2022Black-box Few-shot Knowledge Distillation
  3. 2021A Fast Knowledge Distillation Framework for Visual Recognition
  4. 2023CrossKD: Cross-Head Knowledge Distillation for Object Detection
  5. 2021Similarity Transfer for Knowledge Distillation
  6. 2023Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learning
  7. 2025Context-Aware Knowledge Distillation with Adaptive Weighting for Image Classification
  8. 2023Knowledge Diffusion for Distillation