mlsanityData to 14 Oct 2026

Switchable Online Knowledge Distillation

arXiv 2209.04996 ↗First posted 12 Sep 2022cs.CVcited by 29

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Online Knowledge Distillation (OKD) improves the involved models by reciprocally exploiting the difference between teacher and student. Several crucial bottlenecks over the gap between them -- e.g., Why and when does a large gap harm the performance, especially for student? How to quantify the gap between teacher and student? -- have received limited formal study. In this paper, we propose Switchable Online Knowledge Distillation (SwitOKD), to answer these questions. Instead of focusing on the accuracy gap at test phase by the existing arts, the core idea of SwitOKD is to adaptively calibrate the gap at training phase, namely distillation gap, via a switching strategy between two modes -- expert mode (pause the teacher while keep the student learning) and learning mode (restart the teacher). To possess an appropriate distillation gap, we further devise an adaptive switching threshold, which provides a formal criterion as to when to switch to learning mode or expert mode, and thus improves the student's performance. Meanwhile, the teacher benefits from our adaptive switching threshold and keeps basically on a par with other online arts. We further extend SwitOKD to multiple networks with two basis topologies. Finally, extensive experiments and analysis validate the merits of SwitOKD for classification over the state-of-the-arts. Our code is available at https://github.com/hfutqian/SwitOKD.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 24 of its 33 references that are in the map. First posted on arXiv 12 Sep 2022.

Builds on

24 of 33 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Revisiting Knowledge Distillation: An Inheritance and Exploration Framework1 Jul
  2. 2021Distilling Knowledge via Knowledge Review19 Apr
  3. 2021Complementary Relation Contrastive Distillation29 Mar

Other references · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  5. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  6. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  7. 2016Wide Residual Networkscited by 3,070
  8. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  9. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  10. 2019Contrastive Representation Distillationcited by 748
  11. 2017Deep Mutual Learningcited by 628
  12. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
9 more
  1. 2019Similarity-Preserving Knowledge Distillationcited by 516
  2. 2019On the Efficacy of Knowledge Distillationcited by 381
  3. 2020Knowledge Distillation Meets Self-Supervisioncited by 142
  4. 2019Online Knowledge Distillation with Diverse Peerscited by 128
  5. 2019Knowledge Distillation via Route Constrained Optimizationcited by 75
  6. 2020Heterogeneous Knowledge Distillation using Information Flow Modelingcited by 73
  7. 2018Collaborative Learning for Deep Neural Networkscited by 69
  8. 2020Feature-map-level Online Adversarial Knowledge Distillationcited by 62
  9. 2020Survey on Deep Multi-modal Data Analytics: Collaboration, Rivalry and Fusioncited by 53

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 11; 2024: 9; 2025: 4; 2026: 3.

29 papers.

Sort:

By area:

  1. 2023M3Net: Multi-view Encoding, Matching, and Fusion for Few-shot Fine-grained Action Recognitioncited by 34
  2. 2023Adaptive Data-Free Quantizationcited by 26
  3. 2023Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?cited by 21
  4. 2023Rethinking Data-Free Quantization as a Zero-Sum Gamecited by 13
  5. 2024Towards Robust and Efficient Cloud-Edge Elastic Model Adaptation via Selective Entropy Distillationcited by 11
  6. 2024A Comprehensive Review of Knowledge Distillation in Computer Visioncited by 10
  7. 2023Cross Architecture Distillation for Face Recognitioncited by 9
  8. 2024Spectral Co-Distillation for Personalized Federated Learningcited by 8
  9. 2022BD-KD: Balancing the Divergences for Online Knowledge Distillationcited by 7
  10. 2023Decouple Graph Neural Networks: Train Multiple Simple GNNs Simultaneously Instead of Onecited by 7
  11. 2023Learning Comprehensive Representations with Richer Self for Text-to-Image Person Re-Identificationcited by 7
  12. 2023Fine-grained Cross-modal Fusion based Refinement for Text-to-Image Synthesiscited by 6
17 more
  1. 2024CNN-Transformer Rectified Collaborative Learning for Medical Image Segmentationcited by 6
  2. 2022ADPS: Asymmetric Distillation Post-Segmentation for Image Anomaly Detectioncited by 5
  3. 2024Manifold-based Incomplete Multi-view Clustering via Bi-Consistency Guidancecited by 5
  4. 2023Graph-Collaborated Auto-Encoder Hashing for Multi-view Binary Clusteringcited by 4
  5. 2024FedDr+: Stabilizing Dot-regression with Global Feature Distillation for Federated Learningcited by 2
  6. 2023Multi-View Fusion and Distillation for Subgrade Distresses Detection based on 3D-GPRcited by 1
  7. 2024CrimeAlarm: Towards Intensive Intent Dynamics in Fine-grained Crime Predictioncited by 1
  8. 2024Collaborative Knowledge Distillation via a Learning-by-Education Node Communitycited by 1
  9. 2024Map-Free Trajectory Prediction with Map Distillation and Hierarchical Encodingcited by 1
  10. 2025Right Time to Learn:Promoting Generalization via Bio-inspired Spacing Effect in Knowledge Distillationcited by 1
  11. 2025Asymmetric Decision-Making in Online Knowledge Distillation:Unifying Consensus and Divergencecited by 1
  12. 2026Selective Coupling of Decoupled Informative Regions: Masked Attention Alignment for Data-Free Quantization of Vision Transformerscited by 1
  13. 2023Domain Adaptive Person Search via GAN-based Scene Synthesis for Cross-scene Videoscited by 0
  14. 2025AgentPose: Progressive Distribution Alignment via Feature Agent for Human Pose Distillationcited by 0
  15. 2025JointDistill: Adaptive Multi-Task Distillation for Joint Depth Estimation and Scene Segmentationcited by 0
  16. 2026AI-T2I: Aggregating-and-Isolating Cross-Attention to Diffusion Models for Text-to-Image Synthesiscited by 0
  17. 2026STEDiff: Strengthening Text Embedding for Text-to-Image Alignment in Diffusion Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Parameter-Efficient and Student-Friendly Knowledge Distillation
  2. 2026PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning
  3. 2023Improving Knowledge Distillation via Transferring Learning Ability
  4. 2026Integrating Knowledge Distillation Methods: A Sequential Multi-Stage Framework
  5. 2022Knowledge Condensation Distillation
  6. 2024Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher
  7. 2026Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall
  8. 2021Dynamic Knowledge Distillation for Pre-trained Language Models