# Knowledge Distillation approach towards Melanoma Detection

arXiv: https://arxiv.org/abs/2210.08086 · HTML: https://mlsanity.com/paper/2210.08086/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation.
- Builds on 21 of its 58 references that are in the map.
- First posted on arXiv 14 Oct 2022.

## Placed in

- Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/)

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021)
- [Efficient Medical Image Segmentation Based on Knowledge Distillation](https://mlsanity.com/paper/2108.09987/) (2021)
- [Towards Cross-modality Medical Image Segmentation with Online Mutual Knowledge Distillation](https://mlsanity.com/paper/2010.01532/) (2020)
- [Skin Lesion Synthesis with Generative Adversarial Networks](https://mlsanity.com/paper/1902.03253/) (2019)
- [Skin Lesion Analysis Towards Melanoma Detection Using Deep Learning Network](https://mlsanity.com/paper/1703.00577/) (2017)
- [Melanoma Detection using Adversarial Training and Deep Transfer Learning](https://mlsanity.com/paper/2004.06824/) (2020)
- [Quantifying the Value of Lateral Views in Deep Learning for Chest X-rays](https://mlsanity.com/paper/2002.02582/) (2020)
- [Knowledge Distillation with Adaptive Asymmetric Label Sharpening for Semi-supervised Fracture Detection in Chest X-rays](https://mlsanity.com/paper/2012.15359/) (2020)
- [Recovering from Random Pruning: On the Plasticity of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1801.10447/) (2018)
- [Learning Lightweight Pedestrian Detector with Hierarchical Knowledge Distillation](https://mlsanity.com/paper/1909.09325/) (2019)
- [Deep Learning-Based Sentiment Analysis of COVID-19 Vaccination Responses from Twitter Data](https://mlsanity.com/paper/2209.12604/) (2022)

## Built on by (most cited first, 3 of 3)

- [Leveraging Different Learning Styles for Improved Knowledge Distillation in Biomedical Imaging](https://mlsanity.com/paper/2212.02931/) (2022)
- [FedBrain-Distill: Communication-Efficient Federated Brain Tumor Classification Using Ensemble Knowledge Distillation on Non-IID Data](https://mlsanity.com/paper/2409.05359/) (2024)
- [Semi-Supervised Learning with Online Knowledge Distillation for Skin Lesion Classification](https://mlsanity.com/paper/2508.11511/) (2025)

## Abstract (arXiv)

Melanoma is regarded as the most threatening among all skin cancers. There is a pressing need to build systems which can aid in the early detection of melanoma and enable timely treatment to patients. Recent methods are geared towards machine learning based systems where the task is posed as image recognition, tag dermoscopic images of skin lesions as melanoma or non-melanoma. Even though these methods show promising results in terms of accuracy, they are computationally quite expensive to train, that questions the ability of these models to be deployable in a clinical setting or memory constraint devices. To address this issue, we focus on building simple and performant models having few layers, less than ten compared to hundreds. As well as with fewer learnable parameters, 0.26 million (M) compared to 42.5M using knowledge distillation with the goal to detect melanoma from dermoscopic images. First, we train a teacher model using a ResNet-50 to detect melanoma. Using the teacher model, we train the student model known as Distilled Student Network (DSNet) which has around 0.26M parameters using knowledge distillation achieving an accuracy of 91.7%. We compare against ImageNet pre-trained models such MobileNet, VGG-16, Inception-V3, EfficientNet-B0, ResNet-50 and ResNet-101. We find that our approach works well in terms of inference runtime compared to other pre-trained models, 2.57 seconds compared to 14.55 seconds. We find that DSNet (0.26M parameters), which is 15 times smaller, consistently performs better than EfficientNet-B0 (4M parameters) in both melanoma and non-melanoma detection across Precision, Recall and F1 scores