# Boosting Dermatoscopic Lesion Segmentation via Diffusion Models with Visual and Textual Prompts

arXiv: https://arxiv.org/abs/2310.02906 · HTML: https://mlsanity.com/paper/2310.02906/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 13 of its 19 references that are in the map.
- First posted on arXiv 4 Oct 2023.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 13 of 13)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [NIPS 2016 Tutorial: Generative Adversarial Networks](https://mlsanity.com/paper/1701.00160/) (2016)
- [Medical Image Synthesis for Data Augmentation and Anonymization using Generative Adversarial Networks](https://mlsanity.com/paper/1807.10225/) (2018)
- [Label-Free Liver Tumor Segmentation](https://mlsanity.com/paper/2303.14869/) (2023)
- [Early Detection and Localization of Pancreatic Cancer by Label-Free Tumor Synthesis](https://mlsanity.com/paper/2308.03008/) (2023)
- [Synthetic Tumors Make AI Segment Tumors Better](https://mlsanity.com/paper/2210.14845/) (2022)

## Built on by (most cited first, 17 of 17)

- [Towards Generalizable Tumor Synthesis](https://mlsanity.com/paper/2402.19470/) (2024)
- [A Survey on Data Augmentation in Large Model Era](https://mlsanity.com/paper/2401.15422/) (2024)
- [Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?](https://mlsanity.com/paper/2411.03670/) (2024)
- [From Pixel to Cancer: Cellular Automata in Computed Tomography](https://mlsanity.com/paper/2403.06459/) (2024)
- [FreeTumor: Advance Tumor Segmentation via Large-Scale Tumor Synthesis](https://mlsanity.com/paper/2406.01264/) (2024)
- [CAT: Coordinating Anatomical-Textual Prompts for Multi-Organ and Tumor Segmentation](https://mlsanity.com/paper/2406.07085/) (2024)
- [Exploiting Structural Consistency of Chest Anatomy for Unsupervised Anomaly Detection in Radiography Images](https://mlsanity.com/paper/2403.08689/) (2024)
- [Text-Driven Tumor Synthesis](https://mlsanity.com/paper/2412.18589/) (2024)
- [Analyzing Tumors by Synthesis](https://mlsanity.com/paper/2409.06035/) (2024)
- [CT Synthesis with Conditional Diffusion Models for Abdominal Lymph Node Segmentation](https://mlsanity.com/paper/2403.17770/) (2024)
- [Physics-Inspired Generative Models in Medical Imaging: A Review](https://mlsanity.com/paper/2407.10856/) (2024)
- [Expectation-Maximization as the Engine of Scalable Medical Intelligence](https://mlsanity.com/paper/2501.03410/) (2025)
- [See More, Change Less: Anatomy-Aware Diffusion for Contrast Enhancement](https://mlsanity.com/paper/2512.07251/) (2025)
- [Towards Synchronous Memorizability and Generalizability with Site-Modulated Diffusion Replay for Cross-Site Continual Segmentation](https://mlsanity.com/paper/2406.18037/) (2024)
- [ShapeKit](https://mlsanity.com/paper/2506.24003/) (2025)
- [Controllable Skin Synthesis via Lesion-Focused Vector Autoregression Model](https://mlsanity.com/paper/2508.19626/) (2025)
- [AURAD: Anatomy-Pathology Unified Radiology Synthesis with Progressive Representations](https://mlsanity.com/paper/2509.04819/) (2025)

## Abstract (arXiv)

Image synthesis approaches, e.g., generative adversarial networks, have been popular as a form of data augmentation in medical image analysis tasks. It is primarily beneficial to overcome the shortage of publicly accessible data and associated quality annotations. However, the current techniques often lack control over the detailed contents in generated images, e.g., the type of disease patterns, the location of lesions, and attributes of the diagnosis. In this work, we adapt the latest advance in the generative model, i.e., the diffusion model, with the added control flow using lesion-specific visual and textual prompts for generating dermatoscopic images. We further demonstrate the advantage of our diffusion model-based framework over the classical generation models in both the image quality and boosting the segmentation performance on skin lesions. It can achieve a 9% increase in the SSIM image quality measure and an over 5% increase in Dice coefficients over the prior arts.