# Latent Diffusion for Medical Image Segmentation: End to end learning for fast sampling and accuracy

arXiv: https://arxiv.org/abs/2407.12952 · HTML: https://mlsanity.com/paper/2407.12952/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 7 of its 34 references that are in the map.
- First posted on arXiv 17 Jul 2024.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 7 of 7)

- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [MedSegDiff: Medical Image Segmentation with Diffusion Probabilistic Model](https://mlsanity.com/paper/2211.00611/) (2022)
- [MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformer](https://mlsanity.com/paper/2301.11798/) (2023)
- [Multi-Head Attention: Collaborate Instead of Concatenate](https://mlsanity.com/paper/2006.16362/) (2020)
- [Contrastive Conditional Latent Diffusion for Audio-visual Segmentation](https://mlsanity.com/paper/2307.16579/) (2023)
- [HiDiff: Hybrid Diffusion Framework for Medical Image Segmentation](https://mlsanity.com/paper/2407.03548/) (2024)
- [Surf-CDM: Score-Based Surface Cold-Diffusion Model For Medical Image Segmentation](https://mlsanity.com/paper/2312.12649/) (2023)

## Built on by (most cited first, 5 of 5)

- [TextDiffSeg: Text-guided Latent Diffusion Model for 3d Medical Images Segmentation](https://mlsanity.com/paper/2504.11825/) (2025)
- [Not All Pixels Are Equal: Pixel-wise Meta-Learning for Medical Segmentation with Noisy Labels](https://mlsanity.com/paper/2511.18894/) (2025)
- [Diffusion Model in Latent Space for Medical Image Segmentation Task](https://mlsanity.com/paper/2512.01292/) (2025)
- [Generative Latent Representations of 3D Brain MRI for Multi-Task Downstream Analysis in Down Syndrome](https://mlsanity.com/paper/2602.13731/) (2026)
- [Rethinking Diffusion Segmentation: When Does It Rely on Its Noisy State, and Does Diffusion Matter?](https://mlsanity.com/paper/2609.23967/) (2026)

## Abstract (arXiv)

Diffusion Probabilistic Models (DPMs) suffer from inefficient inference due to their slow sampling and high memory consumption, which limits their applicability to various medical imaging applications. In this work, we propose a novel conditional diffusion modeling framework (LDSeg) for medical image segmentation, utilizing the learned inherent low-dimensional latent shape manifolds of the target objects and the embeddings of the source image with an end-to-end framework. Conditional diffusion in latent space not only ensures accurate image segmentation for multiple interacting objects, but also tackles the fundamental issues of traditional DPM-based segmentation methods: (1) high memory consumption, (2) time-consuming sampling process, and (3) unnatural noise injection in the forward and reverse processes. The end-to-end training strategy enables robust representation learning in the latent space related to segmentation features, ensuring significantly faster sampling from the posterior distribution for segmentation generation in the inference phase. Our experiments demonstrate that LDSeg achieved state-of-the-art segmentation accuracy on three medical image datasets with different imaging modalities. In addition, we showed that our proposed model was significantly more robust to noise compared to traditional deterministic segmentation models. The code is available at https://github.com/FahimZaman/LDSeg.git.