# DiffSeg: A Segmentation Model for Skin Lesions Based on Diffusion Difference

arXiv: https://arxiv.org/abs/2404.16474 · HTML: https://mlsanity.com/paper/2404.16474/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 29 references that are in the map.
- First posted on arXiv 25 Apr 2024.

## Builds on (most cited first, 16 of 16)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks](https://mlsanity.com/paper/1807.07356/) (2018)
- [GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models](https://mlsanity.com/paper/2210.02025/) (2022)
- [DeepCut: Object Segmentation from Bounding Box Annotations using Convolutional Neural Networks](https://mlsanity.com/paper/1605.07866/) (2016)
- [Deep Co-Training for Semi-Supervised Image Segmentation](https://mlsanity.com/paper/1903.11233/) (2019)
- [Modality-aware Mutual Learning for Multi-modal Medical Image Segmentation](https://mlsanity.com/paper/2107.09842/) (2021)
- [Semi-Supervised Multi-Organ Segmentation via Deep Multi-Planar Co-Training](https://mlsanity.com/paper/1804.02586/) (2018)
- [CT-Realistic Lung Nodule Simulation from 3D Conditional Generative Adversarial Networks for Robust Lung Segmentation](https://mlsanity.com/paper/1806.04051/) (2018)
- [Annotation-Free Cardiac Vessel Segmentation via Knowledge Transfer from Retinal Images](https://mlsanity.com/paper/1907.11483/) (2019)
- [A deep level set method for image segmentation](https://mlsanity.com/paper/1705.06260/) (2017)

## Built on by (most cited first, 3 of 3)

- [Text-DiFuse: An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model](https://mlsanity.com/paper/2410.23905/) (2024)
- [Scribble-Guided Diffusion for Training-free Text-to-Image Generation](https://mlsanity.com/paper/2409.08026/) (2024)
- [Temporally Consistent Unsupervised Segmentation for Mobile Robot Perception](https://mlsanity.com/paper/2507.22194/) (2025)

## Abstract (arXiv)

Weakly supervised medical image segmentation (MIS) using generative models is crucial for clinical diagnosis. However, the accuracy of the segmentation results is often limited by insufficient supervision and the complex nature of medical imaging. Existing models also only provide a single outcome, which does not allow for the measurement of uncertainty. In this paper, we introduce DiffSeg, a segmentation model for skin lesions based on diffusion difference which exploits diffusion model principles to ex-tract noise-based features from images with diverse semantic information. By discerning difference between these noise features, the model identifies diseased areas. Moreover, its multi-output capability mimics doctors' annotation behavior, facilitating the visualization of segmentation result consistency and ambiguity. Additionally, it quantifies output uncertainty using Generalized Energy Distance (GED), aiding interpretability and decision-making for physicians. Finally, the model integrates outputs through the Dense Conditional Random Field (DenseCRF) algorithm to refine the segmentation boundaries by considering inter-pixel correlations, which improves the accuracy and optimizes the segmentation results. We demonstrate the effectiveness of DiffSeg on the ISIC 2018 Challenge dataset, outperforming state-of-the-art U-Net-based methods.