# A Novel Perspective for Multi-modal Multi-label Skin Lesion Classification

arXiv: https://arxiv.org/abs/2409.12390 · HTML: https://mlsanity.com/paper/2409.12390/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 40 references that are in the map.
- First posted on arXiv 19 Sep 2024.

## Builds on (most cited first, 14 of 14)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [Asymmetric Loss For Multi-Label Classification](https://mlsanity.com/paper/2009.14119/) (2020)
- [Multimodal Deep Learning](https://mlsanity.com/paper/2301.04856/) (2023)
- [Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1605.01397/) (2016)
- [Learning to Predict Visual Attributes in the Wild](https://mlsanity.com/paper/2106.09707/) (2021)
- [Towards Trustable Skin Cancer Diagnosis via Rewriting Model's Decision](https://mlsanity.com/paper/2303.00885/) (2023)
- [Coherent Concept-based Explanations in Medical Image and Its Application to Skin Lesion Diagnosis](https://mlsanity.com/paper/2304.04579/) (2023)
- [A survey, review, and future trends of skin lesion segmentation and classification](https://mlsanity.com/paper/2208.12232/) (2022)
- [MICA: Towards Explainable Skin Lesion Diagnosis via Multi-Level Image-Concept Alignment](https://mlsanity.com/paper/2401.08527/) (2024)
- [TFormer: A throughout fusion transformer for multi-modal skin lesion diagnosis](https://mlsanity.com/paper/2211.11393/) (2022)
- [SSD-KD: A Self-supervised Diverse Knowledge Distillation Method for Lightweight Skin Lesion Classification Using Dermoscopic Images](https://mlsanity.com/paper/2203.11490/) (2022)

## Built on by (most cited first, 2 of 2)

- [Towards a Medical AI Scientist](https://mlsanity.com/paper/2603.28589/) (2026)
- [JI-ADF: Joint-Individual Learning with Adaptive Decision Fusion for Multimodal Skin Lesion Classification](https://mlsanity.com/paper/2604.27343/) (2026)

## Abstract (arXiv)

The efficacy of deep learning-based Computer-Aided Diagnosis (CAD) methods for skin diseases relies on analyzing multiple data modalities (i.e., clinical+dermoscopic images, and patient metadata) and addressing the challenges of multi-label classification. Current approaches tend to rely on limited multi-modal techniques and treat the multi-label problem as a multiple multi-class problem, overlooking issues related to imbalanced learning and multi-label correlation. This paper introduces the innovative Skin Lesion Classifier, utilizing a Multi-modal Multi-label TransFormer-based model (SkinM2Former). For multi-modal analysis, we introduce the Tri-Modal Cross-attention Transformer (TMCT) that fuses the three image and metadata modalities at various feature levels of a transformer encoder. For multi-label classification, we introduce a multi-head attention (MHA) module to learn multi-label correlations, complemented by an optimisation that handles multi-label and imbalanced learning problems. SkinM2Former achieves a mean average accuracy of 77.27% and a mean diagnostic accuracy of 77.85% on the public Derm7pt dataset, outperforming state-of-the-art (SOTA) methods.