mlsanityData to 14 Oct 2026

Leveraging Knowledge Distillation for Lightweight Skin Cancer Classification: Balancing Accuracy and Computational Efficiency

arXiv 2406.17051 ↗First posted 24 Jun 2024eess.IV cs.CV cs.LGcited by 0

On the map: Knowledge distillation → Distillation for medical imaging

Abstract (arXiv)

Skin cancer is a major concern to public health, accounting for one-third of the reported cancers. If not detected early, the cancer has the potential for severe consequences. Recognizing the critical need for effective skin cancer classification, we address the limitations of existing models, which are often too large to deploy in areas with limited computational resources. In response, we present a knowledge distillation based approach for creating a lightweight yet high-performing classifier. The proposed solution involves fusing three models, namely ResNet152V2, ConvNeXtBase, and ViT Base, to create an effective teacher model. The teacher model is then employed to guide a lightweight student model of size 2.03 MB. This student model is further compressed to 469.77 KB using 16-bit quantization, enabling smooth incorporation into edge devices. With six-stage image preprocessing, data augmentation, and a rigorous ablation study, the model achieves an impressive accuracy of 98.75% on the HAM10000 dataset and 98.94% on the Kaggle dataset in classifying benign and malignant skin cancers. With its high accuracy and compact size, our model appears to be a potential choice for accurate skin cancer classification, particularly in resource-constrained settings.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation. Builds on 14 of its 52 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

14 of 52 references.

References · 14

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2017Squeeze-and-Excitation Networkscited by 4,789
  6. 2022A ConvNet for the 2020scited by 3,579
  7. 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 745
  8. 2021Knowledge distillation: A good teacher is patient and consistentcited by 259
  9. 2022Transformers in Medical Imaging: A Surveycited by 245
  10. 2023Semantic Segmentation using Vision Transformers: A surveycited by 73
  11. 2022A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classificationcited by 65
  12. 2022A Review of Deep Transfer Learning and Recent Advancementscited by 57
2 more
  1. 2022A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biasescited by 30
  2. 2022Spot-adaptive Knowledge Distillationcited by 22

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022SSD-KD: A Self-supervised Diverse Knowledge Distillation Method for Lightweight Skin Lesion Classification Using Dermoscopic Images
  2. 2022Knowledge Distillation approach towards Melanoma Detection
  3. 2023SkinDistilViT: Lightweight Vision Transformer for Skin Lesion Classification
  4. 2026DerMAE: Improving skin lesion classification through conditioned latent diffusion and MAE distillation
  5. 2025DCSNet: A Lightweight Knowledge Distillation-Based Model with Explainable AI for Lung Cancer Diagnosis from Histopathological Images
  6. 2021Resolution-Based Distillation for Efficient Histology Image Classification
  7. 2026CNNs, Transformers, Hybrid, and Vision Language Models for Skin Cancer Detection
  8. 2024Self-supervised learning for skin cancer diagnosis with limited training data