# Generative Medical Segmentation

arXiv: https://arxiv.org/abs/2403.18198 · HTML: https://mlsanity.com/paper/2403.18198/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 27 Mar 2024.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Gland Segmentation in Colon Histology Images: The GlaS Challenge Contest](https://mlsanity.com/paper/1603.00275/) (2016)
- [UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer](https://mlsanity.com/paper/2109.04335/) (2021)
- [MultiResUNet : Rethinking the U-Net Architecture for Multimodal Biomedical Image Segmentation](https://mlsanity.com/paper/1902.04049/) (2019)
- [Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization](https://mlsanity.com/paper/2104.05833/) (2021)
- [Uncertainty Modeling for Out-of-Distribution Generalization](https://mlsanity.com/paper/2202.03958/) (2022)
- [MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformer](https://mlsanity.com/paper/2301.11798/) (2023)
- [Breaking the Dilemma of Medical Image-to-image Translation](https://mlsanity.com/paper/2110.06465/) (2021)
- [MixStyle Neural Networks for Domain Generalization and Adaptation](https://mlsanity.com/paper/2107.02053/) (2021)
- [Kvasir-Instrument: Diagnostic and therapeutic tool segmentation dataset in gastrointestinal endoscopy](https://mlsanity.com/paper/2011.08065/) (2020)
- [EGE-UNet: an Efficient Group Enhanced UNet for skin lesion segmentation](https://mlsanity.com/paper/2307.08473/) (2023)
- [Generative Semantic Segmentation](https://mlsanity.com/paper/2303.11316/) (2023)
- [BerDiff: Conditional Bernoulli Diffusion Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.04429/) (2023)
- [ACC-UNet: A Completely Convolutional UNet model for the 2020s](https://mlsanity.com/paper/2308.13680/) (2023)
- [Semi-supervised Task-driven Data Augmentation for Medical Image Segmentation](https://mlsanity.com/paper/2007.05363/) (2020)
- [Synthetic Augmentation with Large-scale Unconditional Pre-training](https://mlsanity.com/paper/2308.04020/) (2023)
- [Brain Lesion Synthesis via Progressive Adversarial Variational Auto-Encoder](https://mlsanity.com/paper/2208.03203/) (2022)

## Built on by (most cited first, 6 of 6)

- [SegMoTE: Token-Level Mixture of Experts for Medical Image Segmentation](https://mlsanity.com/paper/2602.19213/) (2026)
- [SDiFL: Stable Diffusion-Driven Framework for Image Forgery Localization](https://mlsanity.com/paper/2508.20182/) (2025)
- [Wave-GMS: Lightweight Multi-Scale Generative Model for Medical Image Segmentation](https://mlsanity.com/paper/2510.03216/) (2025)
- [Cell Instance Segmentation via Multi-Task Image-to-Image Schr"odinger Bridge](https://mlsanity.com/paper/2604.12318/) (2026)
- [SemiGDA: Generative Dual-distribution Alignment for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2604.23274/) (2026)
- [GET: Generative Embedding Translation for Medical Image Segmentation](https://mlsanity.com/paper/2608.22619/) (2026)

## Abstract (arXiv)

Rapid advancements in medical image segmentation performance have been significantly driven by the development of Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). These models follow the discriminative pixel-wise classification learning paradigm and often have limited ability to generalize across diverse medical imaging datasets. In this manuscript, we introduce Generative Medical Segmentation (GMS), a novel approach leveraging a generative model to perform image segmentation. Concretely, GMS employs a robust pre-trained vision foundation model to extract latent representations for images and corresponding ground truth masks, followed by a model that learns a mapping function from the image to the mask in the latent space. Once trained, the model generates an estimated segmentation mask using the pre-trained vision foundation model to decode the predicted latent representation back into the image space. The design of GMS leads to fewer trainable parameters in the model which reduces the risk of overfitting and enhances its generalization capability. Our experimental analysis across five public datasets in different medical imaging domains demonstrates GMS outperforms existing discriminative and generative segmentation models. Furthermore, GMS is able to generalize well across datasets from different centers within the same imaging modality. Our experiments suggest GMS offers a scalable and effective solution for medical image segmentation. GMS implementation and trained model weights are available at https://github.com/King-HAW/GMS.