mlsanityData to 7 Oct 2026

A Simple Latent Diffusion Approach for Panoptic Segmentation and Mask Inpainting

arXiv 2401.10227 ↗First posted 18 Jan 2024cs.CV cs.LGcited by 18

On the map: Diffusion models → Medical imaging and segmentation

Abstract (arXiv)

Panoptic and instance segmentation networks are often trained with specialized object detection modules, complex loss functions, and ad-hoc post-processing steps to manage the permutation-invariance of the instance masks. This work builds upon Stable Diffusion and proposes a latent diffusion approach for panoptic segmentation, resulting in a simple architecture that omits these complexities. Our training consists of two steps: (1) training a shallow autoencoder to project the segmentation masks to latent space; (2) training a diffusion model to allow image-conditioned sampling in latent space. This generative approach unlocks the exploration of mask completion or inpainting. The experimental validation on COCO and ADE20k yields strong segmentation results. Finally, we demonstrate our model's adaptability to multi-tasking by introducing learnable task embeddings.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. Builds on 81 of its 94 references that are in the map. First posted on arXiv 18 Jan 2024.

Builds on

81 of 94 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models8 Mar
  2. 2022DiffusionInst: Diffusion Model for Instance Segmentation6 Dec
  3. 2021Label-Efficient Semantic Segmentation with Diffusion Models6 Dec
  4. 2021SegDiff: Image Segmentation with Diffusion Probabilistic Models1 Dec

Other references · 77

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  10. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  11. 2020Denoising Diffusion Implicit Modelscited by 9,655
  12. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
38 more
  1. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  2. 2023Segment Anythingcited by 8,158
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2017Mask R-CNNcited by 7,259
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  12. 2017Neural Discrete Representation Learningcited by 4,650
  13. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  14. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  15. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  16. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  17. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  18. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  19. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  20. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  21. 2023Consistency Modelscited by 1,778
  22. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  23. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  24. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  25. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  26. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  27. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  28. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  29. 2016Pixel Recurrent Neural Networkscited by 814
  30. 2018Panoptic Segmentationcited by 796
  31. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  32. 2020Image Segmentation Using Deep Learning: A Surveycited by 659
  33. 2019Panoptic Feature Pyramid Networkscited by 644
  34. 2017Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentationscited by 614
  35. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  36. 2019PointRend: Image Segmentation as Renderingcited by 439
  37. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  38. 2017Soft-NMS -- Improving Object Detection With One Line of Codecited by 380

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 3; 2026: 5.

18 papers.

Sort:

By area:

  1. 2024What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?cited by 74
  2. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  3. 2024LaVin-DiT: Large Vision Diffusion Transformercited by 21
  4. 2024SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flowcited by 20
  5. 2025PanSt3R: Multi-view Consistent Panoptic Segmentationcited by 19
  6. 2025Unified Dense Prediction of Video Diffusioncited by 13
  7. 2024Explore In-Context Segmentation via Latent Diffusion Modelscited by 12
  8. 2024Generative Edge Detection with Stable Diffusioncited by 5
  9. 2026Diffusion Model as a Generalist Segmentation Learnercited by 4
  10. 2024Investigating the Effectiveness of Cross-Attention to Unlock Zero-Shot Editing of Text-to-Video Diffusion Modelscited by 3
  11. 2024A Simple and Generalist Approach for Panoptic Segmentationcited by 3
  12. 2024Zippo: Zipping Color and Transparency Distributions into a Single Diffusion Modelcited by 2
6 more
  1. 2025More Than Generation: Unifying Generation and Depth Estimation via Text-to-Image Diffusion Modelscited by 2
  2. 2026FMS2: Unified Flow Matching for Segmentation and Synthesis of Thin Structurescited by 2
  3. 2024Guided Synthesis of Labeled Brain MRI Data Using Latent Diffusion Models for Segmentation of Enlarged Ventriclescited by 1
  4. 2026Rethinking Vector Field Learning for Generative Segmentationcited by 0
  5. 2026InstEditSeg: Instruction-Driven Image Editing for Polyp and Skin Lesion Segmentationcited by 0
  6. 2026AHMAD: Adaptive Hybrid Multi-task Vision Learning with Assisted Distillation for Keypoint Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Conditional Latent Diffusion Models for Zero-Shot Instance Segmentation
  2. 2025TRACE: Your Diffusion Model is Secretly an Instance Edge Detector
  3. 2024Outline-Guided Object Inpainting with Diffusion Models
  4. 2024BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion
  5. 2024Panoptic Diffusion Models: co-generation of images and segmentation maps
  6. 2023Denoising Diffusion Semantic Segmentation with Mask Prior Modeling
  7. 2024Diff-VPS: Video Polyp Segmentation via a Multi-task Diffusion Network with Adversarial Temporal Reasoning
  8. 2026Panoptic Scene Program Diffusion Transformer