mlsanityData to 7 Oct 2026

Semi-Online Knowledge Distillation

arXiv 2111.11747 ↗First posted 23 Nov 2021cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation is an effective and stable method for model compression via knowledge transfer. Conventional knowledge distillation (KD) is to transfer knowledge from a large and well pre-trained teacher network to a small student network, which is a one-way process. Recently, deep mutual learning (DML) has been proposed to help student networks learn collaboratively and simultaneously. However, to the best of our knowledge, KD and DML have never been jointly explored in a unified framework to solve the knowledge distillation problem. In this paper, we investigate that the teacher model supports more trustworthy supervision signals in KD, while the student captures more similar behaviors from the teacher in DML. Based on these observations, we first propose to combine KD with DML in a unified framework. Furthermore, we propose a Semi-Online Knowledge Distillation (SOKD) method that effectively improves the performance of the student and the teacher. In this method, we introduce the peer-teaching training fashion in DML in order to alleviate the student's imitation difficulty, and also leverage the supervision signals provided by the well-trained teacher in KD. Besides, we also show our framework can be easily extended to feature-based distillation methods. Extensive experiments on CIFAR-100 and ImageNet datasets demonstrate the proposed method achieves state-of-the-art performance.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 34 of its 45 references that are in the map. First posted on arXiv 23 Nov 2021.

Builds on

34 of 45 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2016Densely Connected Convolutional Networkscited by 6,691
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  8. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  9. 2016Wide Residual Networkscited by 3,057
  10. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  11. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  12. 2014FitNets: Hints for Thin Deep Netscited by 1,619
22 more
  1. 2020Knowledge Distillation: A Surveycited by 1,578
  2. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  3. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  4. 2019Contrastive Representation Distillationcited by 734
  5. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  6. 2017Deep Mutual Learningcited by 621
  7. 2019Similarity-Preserving Knowledge Distillationcited by 507
  8. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  9. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
  10. 2019Variational Information Distillation for Knowledge Transfercited by 321
  11. 2019Correlation Congruence for Knowledge Distillationcited by 280
  12. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 212
  13. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  14. 2018Discrimination-aware Channel Pruning for Deep Neural Networkscited by 171
  15. 2019Learning Lightweight Lane Detection CNNs by Self Attention Distillationcited by 162
  16. 2020Regularizing Class-wise Predictions via Self-knowledge Distillationcited by 144
  17. 2020Knowledge Distillation Meets Self-Supervisioncited by 142
  18. 2020Cross-Layer Distillation with Semantic Calibrationcited by 141
  19. 2019Online Knowledge Distillation with Diverse Peerscited by 127
  20. 2019Towards Efficient Model Compression via Learned Global Rankingcited by 75
  21. 2019Knowledge Distillation via Route Constrained Optimizationcited by 73
  22. 2020Knowledge Transfer via Dense Cross-Layer Mutual-Distillationcited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 1; 2024: 0; 2025: 1; 2026: 4.

8 papers.

  1. 2022Parameter-Efficient and Student-Friendly Knowledge Distillationcited by 47
  2. 2023AGTGAN: Unpaired Image Translation for Photographic Ancient Character Generationcited by 17
  3. 2026MemWeaver: Weaving Hybrid Memories for Traceable Long-Horizon Agentic Reasoningcited by 7
  4. 2021Improved Knowledge Distillation via Adversarial Collaborationcited by 3
  5. 2026Factorized Latent Reasoning for LLM-based Recommendationcited by 3
  6. 2025Gaussian Mixture Flow Matching with Domain Alignment for Multi-Domain Sequential Recommendationcited by 1
  7. 2026Purifying Multimodal Retrieval: Fragment-Level Evidence Selection for RAGcited by 1
  8. 2026Learning to Retrieve Navigable Candidates for Efficient Vision-and-Language Navigationcited by 0