ORC: Network Group-based Knowledge Distillation using Online Role Change
Not on the map. Why?
Abstract (arXiv)
In knowledge distillation, since a single, omnipotent teacher network cannot solve all problems, multiple teacher-based knowledge distillations have been studied recently. However, sometimes their improvements are not as good as expected because some immature teachers may transfer the false knowledge to the student. In this paper, to overcome this limitation and take the efficacy of the multiple networks, we divide the multiple networks into teacher and student groups, respectively. That is, the student group is a set of immature networks that require learning the teacher's knowledge, while the teacher group consists of the selected networks that are capable of teaching successfully. We propose our online role change strategy where the top-ranked networks in the student group are able to promote to the teacher group at every iteration. After training the teacher group using the error samples of the student group to refine the teacher group's knowledge, we transfer the collaborative knowledge from the teacher group to the student group successfully. We verify the superiority of the proposed method on CIFAR-10, CIFAR-100, and ImageNet which achieves high performance. We further show the generality of our method with various backbone architectures such as ResNet, WRN, VGG, Mobilenet, and Shufflenet.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 35 of its 46 references that are in the map. First posted on arXiv 1 Jun 2022.
Builds on
35 of 46 references.
References · 35
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
- 2016Wide Residual Networkscited by 3,057
- 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
- 2016Identity Mappings in Deep Residual Networkscited by 1,960
- 2014FitNets: Hints for Thin Deep Netscited by 1,619
- 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
- 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
- 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
23 more
- 2016Pruning Filters for Efficient ConvNetscited by 1,129
- 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
- 2019Relational Knowledge Distillationcited by 873
- 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
- 2019Contrastive Representation Distillationcited by 734
- 2017Deep Mutual Learningcited by 621
- 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
- 2019Similarity-Preserving Knowledge Distillationcited by 507
- 2022Decoupled Knowledge Distillationcited by 380
- 2019On the Efficacy of Knowledge Distillationcited by 377
- 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 375
- 2019Variational Information Distillation for Knowledge Transfercited by 321
- 2019Correlation Congruence for Knowledge Distillationcited by 280
- 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 255
- 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 235
- 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 212
- 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
- 2020Knowledge Distillation Meets Self-Supervisioncited by 142
- 2019Online Knowledge Distillation with Diverse Peerscited by 127
- 2020Densely Guided Knowledge Distillation using Multiple Teacher Assistantscited by 82
- 2015Compressing Convolutional Neural Networkscited by 25
- 2019A Large RGB-D Dataset for Semi-supervised Monocular Depth Estimationcited by 21
- 2022itKD: Interchange Transfer-based Knowledge Distillation for 3D Object Detectioncited by 14
Built on by
Citing papers per year: 2026: 1.
1 paper.