# Prompting Medical Vision-Language Models to Mitigate Diagnosis Bias by Generating Realistic Dermoscopic Images

arXiv: https://arxiv.org/abs/2504.01838 · HTML: https://mlsanity.com/paper/2504.01838/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 2 Apr 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset](https://mlsanity.com/paper/2104.09957/) (2021)
- [A Patient-Centric Dataset of Images and Metadata for Identifying Melanomas Using Clinical Context](https://mlsanity.com/paper/2008.07360/) (2020)
- [Disparities in Dermatology AI: Assessments Using Diverse Clinical Images](https://mlsanity.com/paper/2111.08006/) (2021)
- [FairPrune: Achieving Fairness Through Pruning for Dermatological Disease Diagnosis](https://mlsanity.com/paper/2203.02110/) (2022)
- [The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions](https://mlsanity.com/paper/1803.10417/) (2018)
- [Detecting Melanoma Fairly: Skin Tone Detection and Debiasing for Skin Lesion Classification](https://mlsanity.com/paper/2202.02832/) (2022)
- [FairAdaBN: Mitigating unfairness with adaptive batch normalization and its application to dermatological disease classification](https://mlsanity.com/paper/2303.08325/) (2023)
- [Toward Fairness Through Fair Multi-Exit Framework for Dermatological Disease Diagnosis](https://mlsanity.com/paper/2306.14518/) (2023)
- [The (de)biasing effect of GAN-based augmentation methods on skin lesion images](https://mlsanity.com/paper/2206.15182/) (2022)
- [Iterative Online Image Synthesis via Diffusion Model for Imbalanced Classification](https://mlsanity.com/paper/2403.08407/) (2024)
- [DermDiff: Generative Diffusion Model for Mitigating Racial Biases in Dermatology Diagnosis](https://mlsanity.com/paper/2503.17536/) (2025)

## Built on by (most cited first, 1 of 1)

- [Class-N-Diff: Classification-Induced Diffusion Model Can Make Fair Skin Cancer Diagnosis](https://mlsanity.com/paper/2510.16887/) (2025)

## Abstract (arXiv)

Artificial Intelligence (AI) in skin disease diagnosis has improved significantly, but a major concern is that these models frequently show biased performance across subgroups, especially regarding sensitive attributes such as skin color. To address these issues, we propose a novel generative AI-based framework, namely, Dermatology Diffusion Transformer (DermDiT), which leverages text prompts generated via Vision Language Models and multimodal text-image learning to generate new dermoscopic images. We utilize large vision language models to generate accurate and proper prompts for each dermoscopic image which helps to generate synthetic images to improve the representation of underrepresented groups (patient, disease, etc.) in highly imbalanced datasets for clinical diagnoses. Our extensive experimentation showcases the large vision language models providing much more insightful representations, that enable DermDiT to generate high-quality images. Our code is available at https://github.com/Munia03/DermDiT