# GS: Generative Segmentation via Label Diffusion

arXiv: https://arxiv.org/abs/2508.20020 · HTML: https://mlsanity.com/paper/2508.20020/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 27 Aug 2025.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 22 of 22)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Your Diffusion Model is Secretly a Zero-Shot Classifier](https://mlsanity.com/paper/2303.16203/) (2023)
- [MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis](https://mlsanity.com/paper/2211.09117/) (2022)
- [Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation](https://mlsanity.com/paper/2003.08813/) (2020)
- [Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion](https://mlsanity.com/paper/2308.12469/) (2023)
- [Dataset Diffusion: Diffusion-based Synthetic Dataset Generation for Pixel-Level Semantic Segmentation](https://mlsanity.com/paper/2309.14303/) (2023)
- [Improving Diffusion Models for Authentic Virtual Try-on in the Wild](https://mlsanity.com/paper/2403.05139/) (2024)
- [Open-vocabulary Object Segmentation with Diffusion Models](https://mlsanity.com/paper/2301.05221/) (2023)
- [DiffusionSeg: Adapting Diffusion Towards Unsupervised Object Discovery](https://mlsanity.com/paper/2303.09813/) (2023)
- [Ref-Diff: Zero-shot Referring Image Segmentation with Generative Models](https://mlsanity.com/paper/2308.16777/) (2023)
- [Panoptic Narrative Grounding](https://mlsanity.com/paper/2109.04988/) (2021)
- [VGDiffZero: Text-to-image Diffusion Models Can Be Zero-shot Visual Grounders](https://mlsanity.com/paper/2309.01141/) (2023)
- [Towards Real-Time Panoptic Narrative Grounding by an End-to-End Grounding Network](https://mlsanity.com/paper/2301.03160/) (2023)
- [Exploring Phrase-Level Grounding with Text-to-Image Diffusion Model](https://mlsanity.com/paper/2407.05352/) (2024)
- [PPMN: Pixel-Phrase Matching Network for One-Stage Panoptic Narrative Grounding](https://mlsanity.com/paper/2208.05647/) (2022)
- [NICE: Improving Panoptic Narrative Detection and Segmentation with Cascading Collaborative Learning](https://mlsanity.com/paper/2310.10975/) (2023)
- [Enriching Phrases with Coupled Pixel and Object Contexts for Panoptic Narrative Grounding](https://mlsanity.com/paper/2311.01091/) (2023)
- [Context Does Matter: End-to-end Panoptic Narrative Grounding with Deformable Attention Refined Matching Network](https://mlsanity.com/paper/2310.16616/) (2023)
- [Dynamic Prompting of Frozen Text-to-Image Diffusion Models for Panoptic Narrative Grounding](https://mlsanity.com/paper/2409.08251/) (2024)

## Built on by (most cited first, 2 of 2)

- [In-Situ Tweedie Discrete Diffusion Models](https://mlsanity.com/paper/2510.01047/) (2025)
- [Prompting Diffusion Models for Zero-Shot Instance Segmentation](https://mlsanity.com/paper/2606.22660/) (2026)

## Abstract (arXiv)

Language-driven image segmentation is a fundamental task in vision-language understanding, requiring models to segment regions of an image corresponding to natural language expressions. Traditional methods approach this as a discriminative problem, assigning each pixel to foreground or background based on semantic alignment. Recently, diffusion models have been introduced to this domain, but existing approaches remain image-centric: they either (i) use image diffusion models as visual feature extractors, (ii) synthesize segmentation data via image generation to train discriminative models, or (iii) perform diffusion inversion to extract attention cues from pre-trained image diffusion models-thereby treating segmentation as an auxiliary process. In this paper, we propose GS (Generative Segmentation), a novel framework that formulates segmentation itself as a generative task via label diffusion. Instead of generating images conditioned on label maps and text, GS reverses the generative process: it directly generates segmentation masks from noise, conditioned on both the input image and the accompanying language description. This paradigm makes label generation the primary modeling target, enabling end-to-end training with explicit control over spatial and semantic fidelity. To demonstrate the effectiveness of our approach, we evaluate GS on Panoptic Narrative Grounding (PNG), a representative and challenging benchmark for multimodal segmentation that requires panoptic-level reasoning guided by narrative captions. Experimental results show that GS significantly outperforms existing discriminative and diffusion-based methods, setting a new state-of-the-art for language-driven segmentation.