# Generative Semantic Segmentation

arXiv: https://arxiv.org/abs/2303.11316 · HTML: https://mlsanity.com/paper/2303.11316/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 62 references that are in the map.
- First posted on arXiv 20 Mar 2023.

## Builds on (most cited first, 30 of 47)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs](https://mlsanity.com/paper/1412.7062/) (2014)

## Built on by (most cited first, 30 of 32)

- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [Image Segmentation in Foundation Model Era: A Survey](https://mlsanity.com/paper/2408.12957/) (2024)
- [Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation](https://mlsanity.com/paper/2312.06462/) (2023)
- [Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulation](https://mlsanity.com/paper/2308.04549/) (2023)
- [DiffTAD: Temporal Action Detection with Proposal Denoising Diffusion](https://mlsanity.com/paper/2303.14863/) (2023)
- [DiffRect: Latent Diffusion Label Rectification for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2407.09918/) (2024)
- [PointSSC: A Cooperative Vehicle-Infrastructure Point Cloud Benchmark for Semantic Scene Completion](https://mlsanity.com/paper/2309.12708/) (2023)
- [CPM: Class-conditional Prompting Machine for Audio-visual Segmentation](https://mlsanity.com/paper/2407.05358/) (2024)
- [Generative Medical Segmentation](https://mlsanity.com/paper/2403.18198/) (2024)
- [Uncertainty-Participation Context Consistency Learning for Semi-supervised Semantic Segmentation](https://mlsanity.com/paper/2412.17331/) (2024)
- [Few-Shot Adaptation of Training-Free Foundation Model for 3D Medical Image Segmentation](https://mlsanity.com/paper/2501.09138/) (2025)
- [LlamaSeg: Image Segmentation via Autoregressive Mask Generation](https://mlsanity.com/paper/2505.19422/) (2025)
- [Unsupervised Segmentation by Diffusing, Walking and Cutting](https://mlsanity.com/paper/2412.04678/) (2024)
- [Self-Supervised Learning for Image Segmentation: A Comprehensive Survey](https://mlsanity.com/paper/2505.13584/) (2025)
- [Not All Pixels Are Equal: Learning Pixel Hardness for Semantic Segmentation](https://mlsanity.com/paper/2305.08462/) (2023)
- [Seg-VAR: Image Segmentation with Visual Autoregressive Modeling](https://mlsanity.com/paper/2511.12594/) (2025)
- [Mitigate Target-level Insensitivity of Infrared Small Target Detection via Posterior Distribution Modeling](https://mlsanity.com/paper/2403.08380/) (2024)
- [Enhancing OOD Detection Using Latent Diffusion](https://mlsanity.com/paper/2406.16525/) (2024)
- [VPNeXt -- Rethinking Dense Decoding for Plain Vision Transformer](https://mlsanity.com/paper/2502.16654/) (2025)
- [Segment Any Vehicle: Semantic and Visual Context Driven SAM and A Benchmark](https://mlsanity.com/paper/2508.04260/) (2025)
- [Re-coding for Uncertainties: Edge-awareness Semantic Concordance for Resilient Event-RGB Segmentation](https://mlsanity.com/paper/2511.08269/) (2025)
- [Synthetic Instance Segmentation from Semantic Image Segmentation Masks](https://mlsanity.com/paper/2308.00949/) (2023)
- [CAM-Seg: A Continuous-valued Embedding Approach for Semantic Image Generation](https://mlsanity.com/paper/2503.15617/) (2025)
- [Multi-modality Anomaly Segmentation on the Road](https://mlsanity.com/paper/2503.17712/) (2025)
- [Wave-GMS: Lightweight Multi-Scale Generative Model for Medical Image Segmentation](https://mlsanity.com/paper/2510.03216/) (2025)
- [Classifier Enhancement Using Extended Context and Domain Experts for Semantic Segmentation](https://mlsanity.com/paper/2510.25174/) (2025)
- [Vision-Language Feature Alignment for Road Anomaly Segmentation](https://mlsanity.com/paper/2603.01029/) (2026)
- [LEMMA: Laplacian pyramids for Efficient Marine SeMAntic Segmentation](https://mlsanity.com/paper/2603.25689/) (2026)
- [SemiGDA: Generative Dual-distribution Alignment for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2604.23274/) (2026)
- [ChangeFlow -- Latent Rectified Flow for Change Detection in Remote Sensing](https://mlsanity.com/paper/2605.15375/) (2026)

## Abstract (arXiv)

We present Generative Semantic Segmentation (GSS), a generative learning approach for semantic segmentation. Uniquely, we cast semantic segmentation as an image-conditioned mask generation problem. This is achieved by replacing the conventional per-pixel discriminative learning with a latent prior learning process. Specifically, we model the variational posterior distribution of latent variables given the segmentation mask. To that end, the segmentation mask is expressed with a special type of image (dubbed as maskige). This posterior distribution allows to generate segmentation masks unconditionally. To achieve semantic segmentation on a given image, we further introduce a conditioning network. It is optimized by minimizing the divergence between the posterior distribution of maskige (i.e., segmentation masks) and the latent prior distribution of input training images. Extensive experiments on standard benchmarks show that our GSS can perform competitively to prior art alternatives in the standard semantic segmentation setting, whilst achieving a new state of the art in the more challenging cross-domain setting.