# Derm-T2IM: Harnessing Synthetic Skin Lesion Data via Stable Diffusion Models for Enhanced Skin Disease Classification using ViT and CNN

arXiv: https://arxiv.org/abs/2401.05159 · HTML: https://mlsanity.com/paper/2401.05159/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 19 of its 39 references that are in the map.
- First posted on arXiv 10 Jan 2024.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Generalizing from a Few Examples: A Survey on Few-Shot Learning](https://mlsanity.com/paper/1904.05046/) (2019)
- [Diffusion Models for Medical Image Analysis: A Comprehensive Survey](https://mlsanity.com/paper/2211.07804/) (2022)
- [ResViT: Residual vision transformers for multi-modal medical image synthesis](https://mlsanity.com/paper/2106.16031/) (2021)
- [Medical Image Synthesis for Data Augmentation and Anonymization using Generative Adversarial Networks](https://mlsanity.com/paper/1807.10225/) (2018)
- [GANs for Medical Image Synthesis: An Empirical Study](https://mlsanity.com/paper/2105.05318/) (2021)
- [Diffusion-based Data Augmentation for Skin Disease Classification: Impact Across Original Medical Datasets to Fully Synthetic Images](https://mlsanity.com/paper/2301.04802/) (2023)
- [Sketch guided and progressive growing GAN for realistic and editable ultrasound image synthesis](https://mlsanity.com/paper/2204.06929/) (2022)
- [Medical diffusion on a budget: Textual Inversion for medical image generation](https://mlsanity.com/paper/2303.13430/) (2023)
- [Automatic Lesion Detection System (ALDS) for Skin Cancer Classification Using SVM and Neural Classifiers](https://mlsanity.com/paper/2003.06276/) (2020)

## Built on by (most cited first, 15 of 15)

- [Ctrl-GenAug: Controllable Generative Augmentation for Medical Sequence Classification](https://mlsanity.com/paper/2409.17091/) (2024)
- [Evaluating and Improving the Effectiveness of Synthetic Chest X-Rays for Medical Image Analysis](https://mlsanity.com/paper/2411.18602/) (2024)
- [SkinGEN: an Explainable Dermatology Diagnosis-to-Generation Framework with Interactive Vision-Language Models](https://mlsanity.com/paper/2404.14755/) (2024)
- [Improved Generation of Synthetic Imaging Data Using Feature-Aligned Diffusion](https://mlsanity.com/paper/2410.00731/) (2024)
- [T2ID-CAS: Diffusion Model and Class Aware Sampling to Mitigate Class Imbalance in Neck Ultrasound Anatomical Landmark Detection](https://mlsanity.com/paper/2504.21231/) (2025)
- [DerMAE: Improving skin lesion classification through conditioned latent diffusion and MAE distillation](https://mlsanity.com/paper/2602.19848/) (2026)
- [ChildDiffusion: Unlocking the Potential of Generative AI and Controllable Augmentations for Child Facial Data using Stable Diffusion and Large Language Models](https://mlsanity.com/paper/2406.11592/) (2024)
- [Divergent Domains, Convergent Grading: Enhancing Generalization in Diabetic Retinopathy Grading](https://mlsanity.com/paper/2411.02614/) (2024)
- [LLM4Sweat: A Trustworthy Large Language Model for Hyperhidrosis Support](https://mlsanity.com/paper/2508.15192/) (2025)
- [Controllable Skin Synthesis via Lesion-Focused Vector Autoregression Model](https://mlsanity.com/paper/2508.19626/) (2025)
- [DermAI: Clinical dermatology acquisition through quality-driven image collection for AI classification in mobile](https://mlsanity.com/paper/2511.10367/) (2025)
- [Synthetic Melanoma Image Generation and Evaluation Using Generative Adversarial Networks](https://mlsanity.com/paper/2603.13497/) (2026)
- [DermaFlux: Synthetic Skin Lesion Generation with Rectified Flows for Enhanced Image Classification](https://mlsanity.com/paper/2603.16392/) (2026)
- [LLM-Based Visual Explanation Evaluation Framework for Assessing the Explainability of Facial Skin Disease Classification Models](https://mlsanity.com/paper/2606.16794/) (2026)
- [What Makes a Good Medical Image Tokenizer? Rethinking Reconstruction and Generation in Medical Image Tokenization](https://mlsanity.com/paper/2609.24691/) (2026)

## Abstract (arXiv)

This study explores the utilization of Dermatoscopic synthetic data generated through stable diffusion models as a strategy for enhancing the robustness of machine learning model training. Synthetic data generation plays a pivotal role in mitigating challenges associated with limited labeled datasets, thereby facilitating more effective model training. In this context, we aim to incorporate enhanced data transformation techniques by extending the recent success of few-shot learning and a small amount of data representation in text-to-image latent diffusion models. The optimally tuned model is further used for rendering high-quality skin lesion synthetic data with diverse and realistic characteristics, providing a valuable supplement and diversity to the existing training data. We investigate the impact of incorporating newly generated synthetic data into the training pipeline of state-of-art machine learning models, assessing its effectiveness in enhancing model performance and generalization to unseen real-world data. Our experimental results demonstrate the efficacy of the synthetic data generated through stable diffusion models helps in improving the robustness and adaptability of end-to-end CNN and vision transformer models on two different real-world skin lesion datasets.