# Explainable Melanoma Diagnosis with Contrastive Learning and LLM-based Report Generation

arXiv: https://arxiv.org/abs/2512.06105 · HTML: https://mlsanity.com/paper/2512.06105/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 17 references that are in the map.
- First posted on arXiv 5 Dec 2025.

## Builds on (most cited first, 7 of 7)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [R2GenGPT: Radiology Report Generation with Frozen LLMs](https://mlsanity.com/paper/2309.09812/) (2023)
- [Dermatologist-like explainable AI enhances trust and confidence in diagnosing melanoma](https://mlsanity.com/paper/2303.12806/) (2023)
- [Prompt-Guided Generation of Structured Chest X-Ray Report Using a Pre-trained LLM](https://mlsanity.com/paper/2404.11209/) (2024)
- [Fool Me Once? Contrasting Textual and Visual Explanations in a Clinical Decision-Support Setting](https://mlsanity.com/paper/2410.12284/) (2024)

## Abstract (arXiv)

Deep learning has demonstrated expert-level performance in melanoma classification, positioning it as a powerful tool in clinical dermatology. However, model opacity and the lack of interpretability remain critical barriers to clinical adoption, as clinicians often struggle to trust the decision-making processes of black-box models. To address this gap, we present a Cross-modal Explainable Framework for Melanoma (CEFM) that leverages contrastive learning as the core mechanism for achieving interpretability. Specifically, CEFM maps clinical criteria for melanoma diagnosis-namely Asymmetry, Border, and Color (ABC)-into the Vision Transformer embedding space using dual projection heads, thereby aligning clinical semantics with visual features. The aligned representations are subsequently translated into structured textual explanations via natural language generation, creating a transparent link between raw image data and clinical interpretation. Experiments on public datasets demonstrate 92.79% accuracy and an AUC of 0.961, along with significant improvements across multiple interpretability metrics. Qualitative analyses further show that the spatial arrangement of the learned embeddings aligns with clinicians' application of the ABC rule, effectively bridging the gap between high-performance classification and clinical trust.