mlsanityData to 14 Oct 2026

Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition

arXiv 2207.11518 ↗First posted 23 Jul 2022cs.CV cs.AIcited by 36

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

The teacher-free online Knowledge Distillation (KD) aims to train an ensemble of multiple student models collaboratively and distill knowledge from each other. Although existing online KD methods achieve desirable performance, they often focus on class probabilities as the core knowledge type, ignoring the valuable feature representational information. We present a Mutual Contrastive Learning (MCL) framework for online KD. The core idea of MCL is to perform mutual interaction and transfer of contrastive distributions among a cohort of networks in an online manner. Our MCL can aggregate cross-network embedding information and maximize the lower bound to the mutual information between two networks. This enables each network to learn extra contrastive knowledge from others, leading to better feature representations, thus improving the performance of visual recognition tasks. Beyond the final layer, we extend MCL to intermediate layers and perform an adaptive layer-matching mechanism trained by meta-optimization. Experiments on image classification and transfer learning to visual recognition tasks show that layer-wise MCL can lead to consistent performance gains against state-of-the-art online KD approaches. The superiority demonstrates that layer-wise MCL can guide the network to generate better feature representations. Our code is publicly avaliable at https://github.com/winycg/L-MCL.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 58 of its 81 references that are in the map. First posted on arXiv 23 Jul 2022.

Builds on

58 of 81 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021Knowledge Distillation Using Hierarchical Self-Supervision Augmented Distribution7 Sep
  2. 2021Hierarchical Self-supervised Augmented Knowledge Distillation29 Jul
  3. 2021Distilling a Powerful Student Model via Online Knowledge Distillation26 Mar
  4. 2021Student Network Learning via Evolutionary Knowledge Distillation23 Mar

Other references · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2017Mask R-CNNcited by 7,277
  10. 2016Densely Connected Convolutional Networkscited by 6,720
  11. 2016Wide Residual Networkscited by 3,070
  12. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
38 more
  1. 2020Supervised Contrastive Learningcited by 2,829
  2. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  3. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  4. 2020Knowledge Distillation: A Surveycited by 1,603
  5. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  6. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  7. 2016Colorful Image Colorizationcited by 1,114
  8. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  9. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 920
  10. 2019Relational Knowledge Distillationcited by 889
  11. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  12. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  13. 2017In Defense of the Triplet Loss for Person Re-Identificationcited by 786
  14. 2020Self-supervised Learning: Generative or Contrastivecited by 694
  15. 2017Deep Mutual Learningcited by 628
  16. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  17. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 472
  18. 2015Deep Metric Learning via Lifted Structured Feature Embeddingcited by 449
  19. 2019Correlation Congruence for Knowledge Distillationcited by 285
  20. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 238
  21. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 234
  22. 2020Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identificationcited by 215
  23. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  24. 2020Cross-Layer Distillation with Semantic Calibrationcited by 143
  25. 2020Knowledge Distillation Meets Self-Supervisioncited by 142
  26. 2021SEED: Self-supervised Distillation For Visual Representationcited by 134
  27. 2019Cross-Batch Memory for Embedding Learningcited by 132
  28. 2019Online Knowledge Distillation with Diverse Peerscited by 128
  29. 2022Cross-Image Relational Knowledge Distillation for Semantic Segmentationcited by 118
  30. 2020Video Playback Rate Perception for Self-supervisedSpatio-Temporal Representation Learningcited by 103
  31. 2020Video Cloze Procedure for Self-Supervised Spatio-Temporal Learningcited by 81
  32. 2019Knowledge Distillation via Route Constrained Optimizationcited by 75
  33. 2020Heterogeneous Knowledge Distillation using Information Flow Modelingcited by 73
  34. 2018Collaborative Learning for Deep Neural Networkscited by 69
  35. 2020Feature-map-level Online Adversarial Knowledge Distillationcited by 62
  36. 2020Peer Collaborative Learning for Online Knowledge Distillationcited by 58
  37. 2021Mutual Contrastive Learning for Visual Representation Learningcited by 42
  38. 2021Dense Semantic Contrast for Self-Supervised Visual Representation Learningcited by 24

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 15; 2025: 11; 2026: 5.

36 papers.

Sort:

By area:

  1. 2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 87
  2. 2025Frequency-Aligned Knowledge Distillation for Lightweight Spatiotemporal Forecastingcited by 55
  3. 2024MPQ-DM: Mixed Precision Quantization for Extremely Low Bit Diffusion Modelscited by 21
  4. 2024Advancing Textual Prompt Learning with Anchored Attributescited by 20
  5. 2025Merlin: Multi-View Representation Learning for Robust Multivariate Time Series Forecasting with Unfixed Missing Ratescited by 20
  6. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  7. 2025Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognitioncited by 16
  8. 2024Sinkhorn Distance Minimization for Knowledge Distillationcited by 15
  9. 2025MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillationcited by 12
  10. 2024A Comprehensive Review of Knowledge Distillation in Computer Visioncited by 10
  11. 2025SRKD: Towards Efficient 3D Point Cloud Segmentation via Structure- and Relation-aware Knowledge Distillationcited by 9
  12. 2024CKD: Contrastive Knowledge Distillation from A Sample-wise Perspectivecited by 7
24 more
  1. 2024CNN-Transformer Rectified Collaborative Learning for Medical Image Segmentationcited by 6
  2. 2024Dynamic Contrastive Knowledge Distillation for Efficient Image Restorationcited by 6
  3. 2025Test-Time Distillation for Continual Model Adaptationcited by 5
  4. 2024Data-Driven Estimation of the False Positive Rate of the Bayes Binary Classifier via Soft Labelscited by 4
  5. 2025MuTri: Multi-view Tri-alignment for OCT to OCTA 3D Image Translationcited by 4
  6. 2025Competitive Distillation: A Simple Learning Strategy for Improving Visual Classificationcited by 4
  7. 2023A Survey of Historical Learning: Learning Models with Learning Historycited by 3
  8. 2023Efficient Masked Autoencoders with Self-Consistencycited by 2
  9. 2024Online Policy Distillation with Decision-Attentioncited by 2
  10. 2025Text Embedding Knows How to Quantize Text-Guided Diffusion Modelscited by 2
  11. 2024Adaptive Modality Balanced Online Knowledge Distillation for Brain-Eye-Computer based Dim Object Detectioncited by 1
  12. 2024Prototype-Driven Multi-Feature Generation for Visible-Infrared Person Re-identificationcited by 1
  13. 2024ECG-guided individual identification via PPGcited by 1
  14. 2025Asymmetric Decision-Making in Online Knowledge Distillation:Unifying Consensus and Divergencecited by 1
  15. 2025Generalizing Supervised Contrastive learning: A Projection Perspectivecited by 1
  16. 2026GaitKD: A Universal Decoupled Distillation Framework for Efficient Gait Recognitioncited by 1
  17. 2023Unifying Synergies between Self-supervised Learning and Dynamic Computationcited by 0
  18. 2023Team AcieLee: Technical Report for EPIC-SOUNDS Audio-Based Interaction Recognition Challenge 2023cited by 0
  19. 2024Adaptive Teaching with Shared Classifier for Knowledge Distillationcited by 0
  20. 2024Discriminative and Consistent Representation Distillationcited by 0
  21. 2026Teacher-Guided Student Self-Knowledge Distillation Using Diffusion Modelcited by 0
  22. 2026FSCE: A Target-Aware Frequency-Spatial Collaborative Enhancement Framework for Noise-Resilient SAR ATRcited by 0
  23. 2026Auto-FlexSwitch: Efficient Dynamic Model Merging via Learnable Task Vector Compressioncited by 0
  24. 2026Logical Consistency as a Bridge: Improving LLM Hallucination Detection via Label Constraint Modeling between Responses and Self-Judgmentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learning
  2. 2025Contrastive Representation Distillation via Multi-Scale Feature Decoupling
  3. 2024Aligning in a Compact Space: Contrastive Knowledge Distillation between Heterogeneous Architectures
  4. 2021Estimating and Maximizing Mutual Information for Knowledge Distillation
  5. 2022Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillation
  6. 2025FiGKD: Fine-Grained Knowledge Distillation via High-Frequency Detail Transfer
  7. 2024LAKD-Activation Mapping Distillation Based on Local Learning
  8. 2024VLM-KD: Knowledge Distillation from VLM for Long-Tail Visual Recognition