CORSD: Class-Oriented Relational Self Distillation
On the map: Knowledge distillation → General knowledge distillation methods
Abstract (arXiv)
Knowledge distillation conducts an effective model compression method while holding some limitations:(1) the feature based distillation methods only focus on distilling the feature map but are lack of transferring the relation of data examples; (2) the relational distillation methods are either limited to the handcrafted functions for relation extraction, such as L2 norm, or weak in inter- and intra- class relation modeling. Besides, the feature divergence of heterogeneous teacher-student architectures may lead to inaccurate relational knowledge transferring. In this work, we propose a novel training framework named Class-Oriented Relational Self Distillation (CORSD) to address the limitations. The trainable relation networks are designed to extract relation of structured data input, and they enable the whole model to better classify samples by transferring the relational knowledge from the deepest layer of the model to shallow layers. Besides, auxiliary classifiers are proposed to make relation networks capture class-oriented relation that benefits classification task. Experiments demonstrate that CORSD achieves remarkable improvements. Compared to baseline, 3.8%, 1.5% and 4.5% averaged accuracy boost can be observed on CIFAR100, ImageNet and CUB-200-2011, respectively.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 20 of its 31 references that are in the map. First posted on arXiv 28 Apr 2023.
Builds on
20 of 31 references.
References · 20
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
- 2017Squeeze-and-Excitation Networkscited by 4,789
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
- 2016Wide Residual Networkscited by 3,070
- 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
- 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
- 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
- 2016Identity Mappings in Deep Residual Networkscited by 1,965
- 2014FitNets: Hints for Thin Deep Netscited by 1,653
- 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
8 more
- 2019Relational Knowledge Distillationcited by 889
- 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
- 2017Deep Mutual Learningcited by 628
- 2018Born Again Neural Networkscited by 546
- 2019Similarity-Preserving Knowledge Distillationcited by 516
- 2019Correlation Congruence for Knowledge Distillationcited by 285
- 2020Regularizing Class-wise Predictions via Self-knowledge Distillationcited by 144
- 2020Self-Knowledge Distillation with Progressive Refinement of Targetscited by 107
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Class-relation Knowledge Distillation for Novel Class Discovery
- 2026Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillation
- 2021HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model Compression
- 2025Multi-Level Decoupled Relational Distillation for Heterogeneous Architectures
- 2024Self-Cooperation Knowledge Distillation for Novel Class Discovery
- 2023Deep Collective Knowledge Distillation
- 2021Hierarchical Self-supervised Augmented Knowledge Distillation
- 2021Complementary Relation Contrastive Distillation