# SSD-KD: A Self-supervised Diverse Knowledge Distillation Method for Lightweight Skin Lesion Classification Using Dermoscopic Images

arXiv: https://arxiv.org/abs/2203.11490 · HTML: https://mlsanity.com/paper/2203.11490/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation.
- Builds on 29 of its 65 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Placed in

- Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/)

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [Transfusion: Understanding Transfer Learning for Medical Imaging](https://mlsanity.com/paper/1902.07208/) (2019)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Correlation Congruence for Knowledge Distillation](https://mlsanity.com/paper/1904.01802/) (2019)
- [BCN20000: Dermoscopic Lesions in the Wild](https://mlsanity.com/paper/1908.02288/) (2019)
- [Knowledge Distillation Meets Self-Supervision](https://mlsanity.com/paper/2006.07114/) (2020)
- [Self-supervised Learning is More Robust to Dataset Imbalance](https://mlsanity.com/paper/2110.05025/) (2021)
- [Skin Lesion Classification Using Ensembles of Multi-Resolution EfficientNets with Meta Data](https://mlsanity.com/paper/1910.03910/) (2019)
- [Efficient Medical Image Segmentation Based on Knowledge Distillation](https://mlsanity.com/paper/2108.09987/) (2021)
- [Distilling and Transferring Knowledge via cGAN-generated Samples for Image Classification and Regression](https://mlsanity.com/paper/2104.03164/) (2021)
- [Classification of Diabetic Retinopathy Using Unlabeled Data and Knowledge Distillation](https://mlsanity.com/paper/2009.00982/) (2020)

## Built on by (most cited first, 12 of 12)

- [A survey, review, and future trends of skin lesion segmentation and classification](https://mlsanity.com/paper/2208.12232/) (2022)
- [Label-Efficient Deep Learning in Medical Image Analysis: Challenges and Future Directions](https://mlsanity.com/paper/2303.12484/) (2023)
- [Applications of Knowledge Distillation in Remote Sensing: A Survey](https://mlsanity.com/paper/2409.12111/) (2024)
- [Occlusion-Robust FAU Recognition by Mining Latent Space of Masked Autoencoders](https://mlsanity.com/paper/2212.04029/) (2022)
- [A Novel Perspective for Multi-modal Multi-label Skin Lesion Classification](https://mlsanity.com/paper/2409.12390/) (2024)
- [Reversing Skin Cancer Adversarial Examples by Multiscale Diffusive and Denoising Aggregation Mechanism](https://mlsanity.com/paper/2208.10373/) (2022)
- [Self-Supervised Multi-Modality Learning for Multi-Label Skin Lesion Classification](https://mlsanity.com/paper/2310.18583/) (2023)
- [Multiple Teachers-Meticulous Student: A Domain Adaptive Meta-Knowledge Distillation Model for Medical Image Classification](https://mlsanity.com/paper/2403.11226/) (2024)
- [A Knowledge Distillation-Based Approach to Enhance Transparency of Classifier Models](https://mlsanity.com/paper/2502.15959/) (2025)
- [Leveraging Different Learning Styles for Improved Knowledge Distillation in Biomedical Imaging](https://mlsanity.com/paper/2212.02931/) (2022)
- [Knowledge Distillation for Adaptive MRI Prostate Segmentation Based on Limit-Trained Multi-Teacher Models](https://mlsanity.com/paper/2303.09494/) (2023)
- [Adaptive Teaching with Shared Classifier for Knowledge Distillation](https://mlsanity.com/paper/2406.08528/) (2024)

## Abstract (arXiv)

Skin cancer is one of the most common types of malignancy, affecting a large population and causing a heavy economic burden worldwide. Over the last few years, computer-aided diagnosis has been rapidly developed and make great progress in healthcare and medical practices due to the advances in artificial intelligence. However, most studies in skin cancer detection keep pursuing high prediction accuracies without considering the limitation of computing resources on portable devices. In this case, knowledge distillation (KD) has been proven as an efficient tool to help improve the adaptability of lightweight models under limited resources, meanwhile keeping a high-level representation capability. To bridge the gap, this study specifically proposes a novel method, termed SSD-KD, that unifies diverse knowledge into a generic KD framework for skin diseases classification. Our method models an intra-instance relational feature representation and integrates it with existing KD research. A dual relational knowledge distillation architecture is self-supervisedly trained while the weighted softened outputs are also exploited to enable the student model to capture richer knowledge from the teacher model. To demonstrate the effectiveness of our method, we conduct experiments on ISIC 2019, a large-scale open-accessed benchmark of skin diseases dermoscopic images. Experiments show that our distilled lightweight model can achieve an accuracy as high as 85% for the classification tasks of 8 different skin diseases with minimal parameters and computing requirements. Ablation studies confirm the effectiveness of our intra- and inter-instance relational knowledge integration strategy. Compared with state-of-the-art knowledge distillation techniques, the proposed method demonstrates improved performances for multi-diseases classification on the large-scale dermoscopy database.