# A Multimodal Vision Foundation Model for Clinical Dermatology

arXiv: https://arxiv.org/abs/2410.15038 · HTML: https://mlsanity.com/paper/2410.15038/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2024.

## Built on by (most cited first, 30 of 50)

- [Derm1M: A Million-scale Vision-Language Dataset Aligned with Clinical Ontology Knowledge for Dermatology](https://mlsanity.com/paper/2503.14911/) (2025)
- [A Fully Open and Generalizable Foundation Model for Ultrasound Clinical Applications](https://mlsanity.com/paper/2509.11752/) (2025)
- [FusionFM: Fusing Eye-specific Foundational Models for Optimized Ophthalmic Diagnosis](https://mlsanity.com/paper/2508.11721/) (2025)
- [CleanPatrick: A Benchmark for Image Data Cleaning](https://mlsanity.com/paper/2505.11034/) (2025)
- [Skin Lesion Phenotyping via Nested Multi-modal Contrastive Learning](https://mlsanity.com/paper/2505.23709/) (2025)
- [Towards Trustworthy Dermatology MLLMs: A Benchmark and Multimodal Evaluator for Diagnostic Narratives](https://mlsanity.com/paper/2511.09195/) (2025)
- [A data- and compute-efficient chest X-ray foundation model beyond aggressive scaling](https://mlsanity.com/paper/2602.22843/) (2026)
- [DINOv3 Visual Representations for Blueberry Perception Toward Robotic Harvesting](https://mlsanity.com/paper/2603.02419/) (2026)
- [CHIPS: Efficient CLIP Adaptation via Curvature-aware Hybrid Influence-based Data Selection](https://mlsanity.com/paper/2511.18519/) (2025)
- [Multi-Aspect Knowledge-Enhanced Medical Vision-Language Pretraining with Multi-Agent Data Generation](https://mlsanity.com/paper/2512.03445/) (2025)
- [A Hierarchical Benchmark of Foundation Models for Dermatology](https://mlsanity.com/paper/2601.12382/) (2026)
- [Fall Risk and Gait Analysis in Community-Dwelling Older Adults using World-Spaced 3D Human Mesh Recovery](https://mlsanity.com/paper/2604.11961/) (2026)
- [Are Multimodal LLMs Ready for Clinical Dermatology? A Real-World Evaluation in Dermatology](https://mlsanity.com/paper/2605.04098/) (2026)
- [An Enhanced Privacy-preserving Federated Few-shot Learning Framework for Respiratory Disease Diagnosis](https://mlsanity.com/paper/2507.08050/) (2025)
- [Grounding Multimodal Large Language Models with Quantitative Skin Attributes: A Retrieval Study](https://mlsanity.com/paper/2508.20188/) (2025)
- [ALL-PET: A Low-resource and Low-shot PET Foundation Model in Projection Domain](https://mlsanity.com/paper/2509.09130/) (2025)
- [ViConEx-Med: Visual Concept Explainability via Multi-Concept Token Transformer for Medical Image Analysis](https://mlsanity.com/paper/2510.10174/) (2025)
- [Machine Learning Framework for Thrombosis Risk Prediction in Rotary Blood Pumps](https://mlsanity.com/paper/2512.15761/) (2025)
- [Does medical specialization of VLMs enhance discriminative power?: A comprehensive investigation through feature distribution analysis](https://mlsanity.com/paper/2601.14774/) (2026)
- [When Life Gives You AI, Will You Turn It Into A Market for Lemons? Understanding How Information Asymmetries About AI System Capabilities Affect Market Outcomes and Adoption](https://mlsanity.com/paper/2601.21650/) (2026)
- [Free Lunch in Medical Image Foundation Model Pre-training via Randomized Synthesis and Disentanglement](https://mlsanity.com/paper/2602.12317/) (2026)
- [OrthoDiffusion: A Generalizable Multi-Task Diffusion Foundation Model for Musculoskeletal MRI Interpretation](https://mlsanity.com/paper/2602.20752/) (2026)
- [BRIGHT: A Collaborative Generalist-Specialist Foundation Model for Breast Pathology](https://mlsanity.com/paper/2603.03030/) (2026)
- [MedKCO: Medical Vision-Language Pretraining via Knowledge-Driven Cognitive Orchestration](https://mlsanity.com/paper/2603.09101/) (2026)
- [Composed Vision-Language Retrieval for Skin Cancer Case Search via Joint Alignment of Global and Local Representations](https://mlsanity.com/paper/2603.09108/) (2026)
- [DermaFlux: Synthetic Skin Lesion Generation with Rectified Flows for Enhanced Image Classification](https://mlsanity.com/paper/2603.16392/) (2026)
- [Early Intervention for VFM-based Multimodal Medical Image Classification](https://mlsanity.com/paper/2603.17514/) (2026)
- [Assessing Multimodal Chronic Wound Embeddings with Expert Triplet Agreement](https://mlsanity.com/paper/2603.29376/) (2026)
- [M-IDoL: Information Decomposition for Modality-Specific and Diverse Representation Learning in Medical Foundation Model](https://mlsanity.com/paper/2604.08936/) (2026)
- [Contrastive Semantic Projection: Faithful Neuron Labeling with Contrastive Examples](https://mlsanity.com/paper/2604.22477/) (2026)

## Abstract (arXiv)

Diagnosing and treating skin diseases require advanced visual skills across domains and the ability to synthesize information from multiple imaging modalities. While current deep learning models excel at specific tasks like skin cancer diagnosis from dermoscopic images, they struggle to meet the complex, multimodal requirements of clinical practice. Here, we introduce PanDerm, a multimodal dermatology foundation model pretrained through self-supervised learning on over 2 million real-world skin disease images from 11 clinical institutions across 4 imaging modalities. We evaluated PanDerm on 28 diverse benchmarks, including skin cancer screening, risk stratification, differential diagnosis of common and rare skin conditions, lesion segmentation, longitudinal monitoring, and metastasis prediction and prognosis. PanDerm achieved state-of-the-art performance across all evaluated tasks, often outperforming existing models when using only 10% of labeled data. We conducted three reader studies to assess PanDerm's potential clinical utility. PanDerm outperformed clinicians by 10.2% in early-stage melanoma detection through longitudinal analysis, improved clinicians' skin cancer diagnostic accuracy by 11% on dermoscopy images, and enhanced non-dermatologist healthcare providers' differential diagnosis by 16.5% across 128 skin conditions on clinical photographs. These results demonstrate PanDerm's potential to improve patient care across diverse clinical scenarios and serve as a model for developing multimodal foundation models in other medical specialties, potentially accelerating the integration of AI support in healthcare. The code can be found at https://github.com/SiyuanYan1/PanDerm.