Deep Collective Knowledge Distillation
On the map: Knowledge distillation → General knowledge distillation methods
Abstract (arXiv)
Many existing studies on knowledge distillation have focused on methods in which a student model mimics a teacher model well. Simply imitating the teacher's knowledge, however, is not sufficient for the student to surpass that of the teacher. We explore a method to harness the knowledge of other students to complement the knowledge of the teacher. We propose deep collective knowledge distillation for model compression, called DCKD, which is a method for training student models with rich information to acquire knowledge from not only their teacher model but also other student models. The knowledge collected from several student models consists of a wealth of information about the correlation between classes. Our DCKD considers how to increase the correlation knowledge of classes during training. Our novel method enables us to create better performing student models for collecting knowledge. This simple yet powerful method achieves state-of-the-art performances in many experiments. For example, for ImageNet, ResNet18 trained with DCKD achieves 72.27%, which outperforms the pretrained ResNet18 by 2.52%. For CIFAR-100, the student model of ShuffleNetV1 with DCKD achieves 6.55% higher top-1 accuracy than the pretrained ShuffleNetV1.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 21 of its 26 references that are in the map. First posted on arXiv 18 Apr 2023.
Builds on
21 of 26 references.
Closest ancestors · 1
Same direction, newest first
Other references · 20
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
- 2016Wide Residual Networkscited by 3,070
- 2014FitNets: Hints for Thin Deep Netscited by 1,653
- 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
- 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
- 2019Relational Knowledge Distillationcited by 889
- 2019Contrastive Representation Distillationcited by 748
8 more
- 2017Deep Mutual Learningcited by 628
- 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 464
- 2019A Comprehensive Overhaul of Feature Distillationcited by 348
- 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 305
- 2019Correlation Congruence for Knowledge Distillationcited by 285
- 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 260
- 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
- 2019Online Knowledge Distillation with Diverse Peerscited by 128
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2021Semi-Online Knowledge Distillation
- 2021Augmenting Knowledge Distillation With Peer-To-Peer Mutual Learning For Model Compression
- 2021Collaborative Teacher-Student Learning via Multiple Knowledge Transfer
- 2026Progressive2: A Teacher-Student Progressive Co-Evolving Knowledge Distillation Method for Substantial Model Compression
- 2022InDistill: Information flow-preserving knowledge distillation for model compression
- 2025HPM-KD: Hierarchical Progressive Multi-Teacher Framework for Knowledge Distillation and Efficient Model Compression
- 2021Knowledge Distillation as Semiparametric Inference
- 2021Estimating and Maximizing Mutual Information for Knowledge Distillation