mlsanityData to 7 Oct 2026

UniGS: Unified Representation for Image Generation and Segmentation

arXiv 2312.01985 ↗First posted 4 Dec 2023cs.CVcited by 29

On the map: Diffusion models → Medical imaging and segmentation

Abstract (arXiv)

This paper introduces a novel unified representation of diffusion models for image generation and segmentation. Specifically, we use a colormap to represent entity-level masks, addressing the challenge of varying entity numbers while aligning the representation closely with the image RGB domain. Two novel modules, including the location-aware color palette and progressive dichotomy module, are proposed to support our mask representation. On the one hand, a location-aware palette guarantees the colors' consistency to entities' locations. On the other hand, the progressive dichotomy module can efficiently decode the synthesized colormap to high-quality entity-level masks in a depth-first binary search without knowing the cluster numbers. To tackle the issue of lacking large-scale segmentation training data, we employ an inpainting pipeline and then improve the flexibility of diffusion models across various tasks, including inpainting, image synthesis, referring segmentation, and entity segmentation. Comprehensive experiments validate the efficiency of our approach, demonstrating comparable segmentation mask quality to state-of-the-art and adaptability to multiple tasks. The code will be released at hrefhttps://github.com/qqlu/Entityhttps://github.com/qqlu/Entity.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. Builds on 65 of its 73 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

65 of 73 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion23 Aug
  2. 2023DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models21 Mar
  3. 2022Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentors23 Nov

Other references · 62

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2023Segment Anythingcited by 8,158
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2017Mask R-CNNcited by 7,259
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  10. 2017Neural Discrete Representation Learningcited by 4,650
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  12. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
38 more
  1. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  2. 2016Pyramid Scene Parsing Networkcited by 2,472
  3. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  5. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  6. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  7. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  8. 2019An Introduction to Variational Autoencoderscited by 997
  9. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  10. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  11. 2018Path Aggregation Network for Instance Segmentationcited by 796
  12. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  13. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  14. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  15. 2023AnyDoor: Zero-shot Object-level Image Customizationcited by 411
  16. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  17. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 330
  18. 2023Pix2Video: Video Editing using Image Diffusioncited by 318
  19. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  20. 2021Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contractioncited by 306
  21. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Modelcited by 276
  22. 2021Score-Based Generative Modeling with Critically-Damped Langevin Diffusioncited by 230
  23. 2022BBDM: Image-to-image Translation with Brownian Bridge Diffusion Modelscited by 222
  24. 2022MAT: Mask-Aware Transformer for Large Hole Image Inpaintingcited by 216
  25. 2021Conditional Image Generation with Score-Based Diffusion Modelscited by 195
  26. 2023SegGPT: Segmenting Everything In Contextcited by 193
  27. 2015DRAW: A Recurrent Neural Network For Image Generationcited by 190
  28. 2019Controllable Text-to-Image Generationcited by 187
  29. 2023Geometric Latent Diffusion Models for 3D Molecule Generationcited by 185
  30. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 162
  31. 2022SINE: SINgle Image Editing with Text-to-Image Diffusion Modelscited by 159
  32. 2023DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Modelscited by 122
  33. 2019Large-scale interactive object segmentation with human annotatorscited by 109
  34. 2018Image Generation from Layoutcited by 105
  35. 2022A Generalist Framework for Panoptic Segmentation of Images and Videoscited by 99
  36. 2021D2C: Diffusion-Denoising Models for Few-shot Conditional Generationcited by 96
  37. 2022High-Quality Entity Segmentationcited by 93
  38. 2021Open-World Entity Segmentationcited by 74

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 12; 2026: 7.

29 papers.

Sort:

By area:

  1. 2024What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?cited by 74
  2. 2024ControlVAR: Exploring Controllable Visual Autoregressive Modelingcited by 58
  3. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  4. 2025DiT360: High-Fidelity Panoramic Image Generation via Hybrid Trainingcited by 24
  5. 2024SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flowcited by 20
  6. 2025VideoAuteur: Towards Long Narrative Video Generationcited by 17
  7. 2024Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentationcited by 16
  8. 2024A Unified Framework for 3D Scene Understandingcited by 15
  9. 2024A Simple Approach to Unifying Diffusion-based Conditional Generationcited by 13
  10. 2025Unified Dense Prediction of Video Diffusioncited by 13
  11. 2024Explore In-Context Segmentation via Latent Diffusion Modelscited by 12
  12. 2025D2GS: Depth-and-Density Guided Gaussian Splatting for Stable and Accurate Sparse-View Reconstructioncited by 10
17 more
  1. 2025G4Seg: Generation for Inexact Segmentation Refinement with Diffusion Modelscited by 5
  2. 2025ARGenSeg: Image Segmentation with Autoregressive Image Generation Modelcited by 5
  3. 2024Zippo: Zipping Color and Transparency Distributions into a Single Diffusion Modelcited by 2
  4. 2025VRMDiff: Text-Guided Video Referring Matting Generation of Diffusioncited by 2
  5. 2025E-SAM: Training-Free Segment Every Entity Modelcited by 2
  6. 2025A Comprehensive Survey on Video Scene Parsing:Advances, Challenges, and Prospectscited by 2
  7. 2025Seg-VAR: Image Segmentation with Visual Autoregressive Modelingcited by 2
  8. 2026FMS2: Unified Flow Matching for Segmentation and Synthesis of Thin Structurescited by 2
  9. 2025Sparse Data Diffusion for Scientific Simulations in Biology and Physicscited by 1
  10. 2026SGSoft: Learning Fused Semantic-Geometric Features for 3D Shape Correspondence via Template-Guided Soft Signalscited by 1
  11. 2024No Re-Train, More Gain: Upgrading Backbones with Diffusion model for Pixel-Wise and Weakly-Supervised Few-Shot Segmentationcited by 0
  12. 2025PartStickers: Generating Parts of Objects for Rapid Prototypingcited by 0
  13. 2026Towards Agnostic and Holistic Universal Image Segmentation with Bit Diffusioncited by 0
  14. 2026Rethinking Vector Field Learning for Generative Segmentationcited by 0
  15. 2026Enhancing In-context Panoramic Generation via Geometric-aware Pretrainingcited by 0
  16. 2026Generative Semantic Segmentation via an Observable Semantic-Image Interface and Hierarchical Generator Evidence Alignmentcited by 0
  17. 2026PixelDense: Dense Prediction as Representation Alignment for Pixel Diffusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Diffuse2Seg: Diffusion Models Can Segment Anything Without Supervision
  2. 2026UniGP: Taming Diffusion Transformer for Prior-Preserved Unified Generation and Perception
  3. 2024Panoptic Diffusion Models: co-generation of images and segmentation maps
  4. 2023Factorized Diffusion Architectures for Unsupervised Image Generation and Segmentation
  5. 2025Palette Aligned Image Diffusion
  6. 2024Free-Mask: A Novel Paradigm of Integration Between the Segmentation Diffusion Model and Image Editing
  7. 2023Denoising Diffusion Semantic Segmentation with Mask Prior Modeling
  8. 2024Mask Factory: Towards High-quality Synthetic Data Generation for Dichotomous Image Segmentation