# MM-Skin: Enhancing Dermatology Vision-Language Model with an Image-Text Dataset Derived from Textbooks

arXiv: https://arxiv.org/abs/2505.06152 · HTML: https://mlsanity.com/paper/2505.06152/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 30 of its 50 references that are in the map.
- First posted on arXiv 9 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 30 of 30)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [PathVQA: 30000+ Questions for Medical Visual Question Answering](https://mlsanity.com/paper/2003.10286/) (2020)
- [SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering](https://mlsanity.com/paper/2102.09542/) (2021)
- [Med-Flamingo: a Multimodal Medical Few-shot Learner](https://mlsanity.com/paper/2307.15189/) (2023)
- [BCN20000: Dermoscopic Lesions in the Wild](https://mlsanity.com/paper/1908.02288/) (2019)
- [HuatuoGPT, towards Taming Language Model to Be a Doctor](https://mlsanity.com/paper/2305.15075/) (2023)
- [PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents](https://mlsanity.com/paper/2303.07240/) (2023)
- [Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset](https://mlsanity.com/paper/2104.09957/) (2021)
- [AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models](https://mlsanity.com/paper/2308.15366/) (2023)
- [Disparities in Dermatology AI: Assessments Using Diverse Clinical Images](https://mlsanity.com/paper/2111.08006/) (2021)
- [PAD-UFES-20: a skin lesion dataset composed of patient data and clinical images collected from smartphones](https://mlsanity.com/paper/2007.00478/) (2020)
- [Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Training](https://mlsanity.com/paper/2209.07098/) (2022)
- [Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training](https://mlsanity.com/paper/2105.11333/) (2021)
- [Overcoming Data Limitation in Medical Visual Question Answering](https://mlsanity.com/paper/1909.11867/) (2019)
- [SkinCon: A skin disease dataset densely annotated by domain experts for fine-grained model debugging and analysis](https://mlsanity.com/paper/2302.00785/) (2023)
- [RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysis](https://mlsanity.com/paper/2404.16754/) (2024)
- [MiniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosis](https://mlsanity.com/paper/2407.04106/) (2024)
- [The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions](https://mlsanity.com/paper/1803.10417/) (2018)
- [MM-Retinal: Knowledge-Enhanced Foundational Pretraining with Fundus Image-Text Expertise](https://mlsanity.com/paper/2405.11793/) (2024)
- [GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI](https://mlsanity.com/paper/2411.14522/) (2024)
- [Crowdsourcing Dermatology Images with Google Search Ads: Creating a Real-World Skin Condition Dataset](https://mlsanity.com/paper/2402.18545/) (2024)
- [RepsNet: Combining Vision with Language for Automated Medical Reports](https://mlsanity.com/paper/2209.13171/) (2022)
- [EasyGen: Easing Multimodal Generation with BiDiffuser and LLMs](https://mlsanity.com/paper/2310.08949/) (2023)

## Built on by (most cited first, 14 of 14)

- [From Learning to Unlearning: Biomedical Security Protection in Multimodal Large Language Models](https://mlsanity.com/paper/2508.04192/) (2025)
- [DermoGPT: Open Weights and Open Data for Morphology-Grounded Dermatological Reasoning MLLMs](https://mlsanity.com/paper/2601.01868/) (2026)
- [SkinCLIP-VL: Consistency-Aware Vision-Language Learning for Multimodal Skin Cancer Diagnosis](https://mlsanity.com/paper/2603.21010/) (2026)
- [See-in-Pairs: Reference Image-Guided Comparative Vision-Language Models for Medical Diagnosis](https://mlsanity.com/paper/2506.18140/) (2025)
- [Towards Trustworthy Dermatology MLLMs: A Benchmark and Multimodal Evaluator for Diagnostic Narratives](https://mlsanity.com/paper/2511.09195/) (2025)
- [Anatomical Region-Guided Contrastive Decoding: A Plug-and-Play Strategy for Mitigating Hallucinations in Medical VLMs](https://mlsanity.com/paper/2512.17189/) (2025)
- [A Hierarchical Benchmark of Foundation Models for Dermatology](https://mlsanity.com/paper/2601.12382/) (2026)
- [DerMAE: Improving skin lesion classification through conditioned latent diffusion and MAE distillation](https://mlsanity.com/paper/2602.19848/) (2026)
- [Grounding Multimodal Large Language Models with Quantitative Skin Attributes: A Retrieval Study](https://mlsanity.com/paper/2508.20188/) (2025)
- [DermAI: Clinical dermatology acquisition through quality-driven image collection for AI classification in mobile](https://mlsanity.com/paper/2511.10367/) (2025)
- [ReXInTheWild: A Unified Benchmark for Medical Photograph Understanding](https://mlsanity.com/paper/2603.19517/) (2026)
- [DermAgent: A Self-Reflective Agentic System for Dermatological Image Analysis with Multi-Tool Reasoning and Traceable Decision-Making](https://mlsanity.com/paper/2605.14403/) (2026)
- [CoFiDA-M: Concept-Aware Feature Modulation for Cross-Domain Adaptation with Image-Only Inference](https://mlsanity.com/paper/2605.31591/) (2026)
- [How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective](https://mlsanity.com/paper/2609.36557/) (2026)

## Abstract (arXiv)

Medical vision-language models (VLMs) have shown promise as clinical assistants across various medical fields. However, specialized dermatology VLM capable of delivering professional and detailed diagnostic analysis remains underdeveloped, primarily due to less specialized text descriptions in current dermatology multimodal datasets. To address this issue, we propose MM-Skin, the first large-scale multimodal dermatology dataset that encompasses 3 imaging modalities, including clinical, dermoscopic, and pathological and nearly 10k high-quality image-text pairs collected from professional textbooks. In addition, we generate over 27k diverse, instruction-following vision question answering (VQA) samples (9 times the size of current largest dermatology VQA dataset). Leveraging public datasets and MM-Skin, we developed SkinVL, a dermatology-specific VLM designed for precise and nuanced skin disease interpretation. Comprehensive benchmark evaluations of SkinVL on VQA, supervised fine-tuning (SFT) and zero-shot classification tasks across 8 datasets, reveal its exceptional performance for skin diseases in comparison to both general and medical VLM models. The introduction of MM-Skin and SkinVL offers a meaningful contribution to advancing the development of clinical dermatology VLM assistants. MM-Skin is available at https://github.com/ZwQ803/MM-Skin