# Categorical Relation-Preserving Contrastive Knowledge Distillation for Medical Image Classification arXiv: https://arxiv.org/abs/2107.03225 ยท HTML: https://mlsanity.com/paper/2107.03225/ ## Key facts - Cited by 14 papers in the map (data through 14 Oct 2026). - Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation. - Builds on 18 of its 31 references that are in the map. - First posted on arXiv 7 Jul 2021. ## Placed in - Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/) ## Builds on (most cited first, 18 of 18) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016) - [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015) - [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018) - [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017) - [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019) - [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019) - [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019) - [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017) - [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019) - [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019) - [Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis](https://mlsanity.com/paper/1804.06353/) (2018) - [Learning Lightweight Lane Detection CNNs by Self Attention Distillation](https://mlsanity.com/paper/1908.00821/) (2019) - [Snapshot Distillation: Teacher-Student Optimization in One Generation](https://mlsanity.com/paper/1812.00123/) (2018) - [Semi-supervised Medical Image Classification with Relation-driven Self-ensembling Model](https://mlsanity.com/paper/2005.07377/) (2020) - [Leveraging Undiagnosed Data for Glaucoma Classification with Teacher-Student Learning](https://mlsanity.com/paper/2007.11355/) (2020) - [Distill-to-Label: Weakly Supervised Instance Labeling Using Knowledge Distillation](https://mlsanity.com/paper/1907.12926/) (2019) - [Classification of Diabetic Retinopathy Using Unlabeled Data and Knowledge Distillation](https://mlsanity.com/paper/2009.00982/) (2020) ## Built on by (most cited first, 14 of 14) - [SCPNet: Semantic Scene Completion on Point Cloud](https://mlsanity.com/paper/2303.06884/) (2023) - [Dissecting Self-Supervised Learning Methods for Surgical Computer Vision](https://mlsanity.com/paper/2207.00449/) (2022) - [Towards Efficient 3D Object Detection with Knowledge Distillation](https://mlsanity.com/paper/2205.15156/) (2022) - [WeakMedSAM: Weakly-Supervised Medical Image Segmentation via SAM with Sub-Class Exploration and Prompt Affinity Mining](https://mlsanity.com/paper/2503.04106/) (2025) - [Contrastive Graph Learning for Population-based fMRI Classification](https://mlsanity.com/paper/2203.14044/) (2022) - [Discrepancy-based Active Learning for Weakly Supervised Bleeding Segmentation in Wireless Capsule Endoscopy Images](https://mlsanity.com/paper/2308.05137/) (2023) - [OTCXR: Rethinking Self-supervised Alignment using Optimal Transport for Chest X-ray Analysis](https://mlsanity.com/paper/2404.11868/) (2024) - [DistilCLIP-EEG: Enhancing Epileptic Seizure Detection Through Multi-modal Learning and Knowledge Distillation](https://mlsanity.com/paper/2510.13497/) (2025) - [Multiple Teachers-Meticulous Student: A Domain Adaptive Meta-Knowledge Distillation Model for Medical Image Classification](https://mlsanity.com/paper/2403.11226/) (2024) - [DCSNet: A Lightweight Knowledge Distillation-Based Model with Explainable AI for Lung Cancer Diagnosis from Histopathological Images](https://mlsanity.com/paper/2505.09334/) (2025) - [Judge Like a Real Doctor: Dual Teacher Sample Consistency Framework for Semi-supervised Medical Image Classification](https://mlsanity.com/paper/2411.03041/) (2024) - [TransFair: Transferring Fairness from Ocular Disease Classification to Progression Prediction](https://mlsanity.com/paper/2412.00051/) (2024) - [VLCD: Vision-Language Contrastive Distillation for Accurate and Efficient Automatic Placenta Analysis](https://mlsanity.com/paper/2506.02229/) (2025) - [UnifiedFL: A Dynamic Unified Learning Framework for Equitable Federation](https://mlsanity.com/paper/2510.26350/) (2025) ## Abstract (arXiv) The amount of medical images for training deep classification models is typically very scarce, making these deep models prone to overfit the training data. Studies showed that knowledge distillation (KD), especially the mean-teacher framework which is more robust to perturbations, can help mitigate the over-fitting effect. However, directly transferring KD from computer vision to medical image classification yields inferior performance as medical images suffer from higher intra-class variance and class imbalance. To address these issues, we propose a novel Categorical Relation-preserving Contrastive Knowledge Distillation (CRCKD) algorithm, which takes the commonly used mean-teacher model as the supervisor. Specifically, we propose a novel Class-guided Contrastive Distillation (CCD) module to pull closer positive image pairs from the same class in the teacher and student models, while pushing apart negative image pairs from different classes. With this regularization, the feature distribution of the student model shows higher intra-class similarity and inter-class variance. Besides, we propose a Categorical Relation Preserving (CRP) loss to distill the teacher's relational knowledge in a robust and class-balanced manner. With the contribution of the CCD and CRP, our CRCKD algorithm can distill the relational knowledge more comprehensively. Extensive experiments on the HAM10000 and APTOS datasets demonstrate the superiority of the proposed CRCKD method.