Data to 7 Oct 2026

Deep Mutual Learning

arXiv 1706.00384 ↗First posted 1 Jun 2017cs.CVcited by 621

Not on the map. Why?

Abstract (arXiv)

Model distillation is an effective and widely used technique to transfer knowledge from a teacher to a student network. The typical application is to transfer from a powerful large network or ensemble to a small network, that is better suited to low-memory or fast execution requirements. In this paper, we present a deep mutual learning (DML) strategy where, rather than one way transfer between a static pre-defined teacher and a student, an ensemble of students learn collaboratively and teach each other throughout the training process. Our experiments show that a variety of network architectures benefit from mutual learning and achieve compelling results on CIFAR-100 recognition and Market-1501 person re-identification benchmarks. Surprisingly, it is revealed that no prior powerful teacher network is necessary -- mutual learning of a collection of simple student networks works, and moreover outperforms distillation from a more powerful yet static teacher.

Key facts. Cited by 621 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 142; 2022: 130; 2023: 124; 2024: 103; 2025: 78; 2026: 44.

621 papers.

Sort:

By area:

2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
2022Decoupled Knowledge Distillationcited by 380
2021Federated Learning on Non-IID Data: A Surveycited by 345
2021R-Drop: Regularized Dropout for Neural Networkscited by 202
2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
2022Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentationcited by 177
2021Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Datacited by 152
2022StylizedNeRF: Consistent 3D Scene Stylization as Stylized NeRF via 2D-3D Mutual Learningcited by 140
2022Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learningcited by 118
2022Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning Systemcited by 110
2021Mutual Consistency Learning for Semi-supervised Medical Image Segmentationcited by 107

38 more

2022PP-OCRv3: More Attempts for the Improvement of Ultra Lightweight OCR Systemcited by 105
2021Semi-supervised Left Atrium Segmentation with Mutual Consistency Trainingcited by 104
2021Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillationcited by 104
2022Generative Cooperative Learning for Unsupervised Video Anomaly Detectioncited by 102
2024Logit Standardization in Knowledge Distillationcited by 101
2021Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signalscited by 100
2023Feature Shrinkage Pyramid for Camouflaged Object Detection with Transformerscited by 87
2021SimCVD: Simple Contrastive Voxel-Wise Representation Distillation for Semi-Supervised Medical Image Segmentationcited by 86
2022Learning Affordance Grounding from Exocentric Imagescited by 82
2022Learning with Limited Annotations: A Survey on Deep Semi-Supervised Learning for Medical Image Segmentationcited by 74
2024PromptKD: Unsupervised Prompt Distillation for Vision-Language Modelscited by 72
2021Adaptive Multi-Teacher Multi-level Knowledge Distillationcited by 69
2021Cloth-Changing Person Re-identification from A Single Image with Gait Prediction and Regularizationcited by 69
2022Nested Collaborative Learning for Long-Tailed Visual Recognitioncited by 63
2022Debiased Self-Training for Semi-Supervised Learningcited by 62
2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
2021Annealing Knowledge Distillationcited by 61
2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
2021EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentationcited by 59
2021Learning Student-Friendly Teacher Networks for Knowledge Distillationcited by 58
2023Decoupled Kullback-Leibler Divergence Losscited by 58
2021Decentralized Federated Learning through Proxy Model Sharingcited by 56
2021Online Knowledge Distillation for Efficient Pose Estimationcited by 55
2024UniMatch V2: Pushing the Limit of Semi-Supervised Semantic Segmentationcited by 53
2021Intra-Inter Camera Similarity for Unsupervised Person Re-Identificationcited by 52
2021A Survey on Green Deep Learningcited by 52
2021Farewell to Mutual Information: Variational Distillation for Cross-Modal Person Re-Identificationcited by 50
2023CrossKD: Cross-Head Knowledge Distillation for Object Detectioncited by 50
2021DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimationcited by 48
2022MSDN: Mutually Semantic Distillation Network for Zero-Shot Learningcited by 48
2021Modality-aware Mutual Learning for Multi-modal Medical Image Segmentationcited by 47
2022Parameter-Efficient and Student-Friendly Knowledge Distillationcited by 47
2024FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learningcited by 47
2021Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillationcited by 46
2021EvDistill: Asynchronous Events to End-task Learning via Bidirectional Reconstruction-guided Cross-modal Knowledge Distillationcited by 46
2021Deep Learning for HDR Imaging: State-of-the-Art and Future Trendscited by 45
2022Composed Image Retrieval with Text Feedback via Multi-grained Uncertainty Regularizationcited by 45
2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44

50 most cited of 621 All 621 (JSON)