mlsanityData to 7 Oct 2026

MetaMixer: A Regularization Strategy for Online Knowledge Distillation

arXiv 2303.07951 ↗First posted 14 Mar 2023cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Online knowledge distillation (KD) has received increasing attention in recent years. However, while most existing online KD methods focus on developing complicated model structures and training strategies to improve the distillation of high-level knowledge like probability distribution, the effects of the multi-level knowledge in the online KD are greatly overlooked, especially the low-level knowledge. Thus, to provide a novel viewpoint to online KD, we propose MetaMixer, a regularization strategy that can strengthen the distillation by combining the low-level knowledge that impacts the localization capability of the networks, and high-level knowledge that focuses on the whole image. Experiments under different conditions show that MetaMixer can achieve significant performance gains over state-of-the-art methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 47 references that are in the map. First posted on arXiv 14 Mar 2023.

Builds on

35 of 47 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2017Mask R-CNNcited by 7,259
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  8. 2016Wide Residual Networkscited by 3,057
  9. 2015Fast R-CNNcited by 2,930
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  11. 2016Pyramid Scene Parsing Networkcited by 2,472
  12. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
23 more
  1. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  2. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  3. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  4. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  5. 2019Relational Knowledge Distillationcited by 873
  6. 2019Contrastive Representation Distillationcited by 734
  7. 2017Deep Mutual Learningcited by 621
  8. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  9. 2019Similarity-Preserving Knowledge Distillationcited by 507
  10. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
  11. 2022Decoupled Knowledge Distillationcited by 380
  12. 2019On the Efficacy of Knowledge Distillationcited by 377
  13. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  14. 2019Correlation Congruence for Knowledge Distillationcited by 280
  15. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 235
  16. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  17. 2019Online Knowledge Distillation with Diverse Peerscited by 127
  18. 2021Convolutional Neural Network Pruning with Structural Redundancy Reductioncited by 73
  19. 2020Feature-map-level Online Adversarial Knowledge Distillationcited by 62
  20. 2020Peer Collaborative Learning for Online Knowledge Distillationcited by 58
  21. 2019Learning Filter Basis for Convolutional Neural Network Compressioncited by 57
  22. 2021Mutual Contrastive Learning for Visual Representation Learningcited by 42
  23. 2020Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation from a Blackbox Modelcited by 28

Built on by

Not cited yet.