mlsanityData to 7 Oct 2026

A Generalist Framework for Panoptic Segmentation of Images and Videos

arXiv 2210.06366 ↗First posted 12 Oct 2022cs.CV cs.AI cs.LGcited by 99

Not on the map. Why?

Cited by 54 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Panoptic segmentation assigns semantic and instance ID labels to every pixel of an image. As permutations of instance IDs are also valid solutions, the task requires learning of high-dimensional one-to-many mapping. As a result, state-of-the-art approaches use customized architectures and task-specific loss functions. We formulate panoptic segmentation as a discrete data generation problem, without relying on inductive bias of the task. A diffusion model is proposed to model panoptic masks, with a simple architecture and generic loss function. By simply adding past predictions as a conditioning signal, our method is capable of modeling video (in a streaming setting) and thereby learns to track object instances automatically. With extensive experiments, we demonstrate that our simple approach can perform competitively to state-of-the-art specialist methods in similar settings.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Builds on 63 of its 72 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

63 of 72 references.

References · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2020Denoising Diffusion Implicit Modelscited by 9,655
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  12. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
38 more
  1. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  2. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  3. 2022A ConvNet for the 2020scited by 3,545
  4. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  5. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  6. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,552
  7. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  8. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  9. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  10. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  11. 2021Transformers in Vision: A Surveycited by 1,009
  12. 2018Panoptic Segmentationcited by 796
  13. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  14. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  15. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  16. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  17. 2019Video Instance Segmentationcited by 391
  18. 2019Panoptic-DeepLabcited by 362
  19. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  20. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  21. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  22. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 341
  23. 2020Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentationcited by 326
  24. 2021SegDiff: Image Segmentation with Diffusion Probabilistic Modelscited by 279
  25. 2019MOTS: Multi-Object Tracking and Segmentationcited by 222
  26. 2021K-Net: Towards Unified Image Segmentationcited by 211
  27. 2019Fixing the train-test resolution discrepancycited by 209
  28. 2021Diffusion Models for Implicit Image Segmentation Ensemblescited by 198
  29. 2019UPSNet: A Unified Panoptic Segmentation Networkcited by 175
  30. 2021Mask2Former for Video Instance Segmentationcited by 169
  31. 2022A Unified Sequence Interface for Vision Taskscited by 152
  32. 2020Video Panoptic Segmentationcited by 141
  33. 2021A Survey on Deep Learning Technique for Video Segmentationcited by 132
  34. 2019Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagationcited by 125
  35. 2020STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videoscited by 121
  36. 2020MAST: A Memory-Augmented Self-supervised Trackercited by 109
  37. 2021Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformerscited by 109
  38. 2020SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentationcited by 108

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 41; 2024: 28; 2025: 19; 2026: 8.

99 papers.

Sort:

By area:

  1. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  2. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  3. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  4. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  5. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 297
  6. 2023A Survey on Video Diffusion Modelscited by 203
  7. 2023On the Importance of Noise Scheduling for Diffusion Modelscited by 193
  8. 2022Scalable Adaptive Computation for Iterative Generationcited by 190
  9. 2023DDP: Diffusion Model for Dense Visual Predictioncited by 148
  10. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  11. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  12. 2023The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimationcited by 116
38 more
  1. 2023Monocular Depth Estimation using Diffusion Modelscited by 87
  2. 2023DiffusionDepth: Diffusion Denoising Approach for Monocular Depth Estimationcited by 81
  3. 2023Diffusion-Based 3D Human Pose Estimation with Multi-Hypothesis Aggregationcited by 81
  4. 2024DepthFM: Fast Monocular Depth Estimation with Flow Matchingcited by 76
  5. 2022DiffusionInst: Diffusion Model for Instance Segmentationcited by 49
  6. 2024Schedule Your Edit: A Simple yet Effective Diffusion Noise Schedule for Image Editingcited by 38
  7. 2023DiffPose: SpatioTemporal Diffusion Model for Video-Based Human Pose Estimationcited by 36
  8. 2023Towards Generic Semi-Supervised Framework for Volumetric Medical Image Segmentationcited by 35
  9. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  10. 2023Generative Semantic Segmentationcited by 32
  11. 2023DiffBEV: Conditional Diffusion Model for Bird's Eye View Perceptioncited by 30
  12. 2023Contrastive Conditional Latent Diffusion for Audio-visual Segmentationcited by 30
  13. 2023FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimatorscited by 29
  14. 2023UniGS: Unified Representation for Image Generation and Segmentationcited by 29
  15. 2023Zero-Shot Metric Depth with a Field-of-View Conditioned Diffusion Modelcited by 29
  16. 2023Parallel Vertex Diffusion for Unified Visual Groundingcited by 28
  17. 2023SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Processcited by 28
  18. 2024OccGen: Generative Multi-modal 3D Occupancy Prediction for Autonomous Drivingcited by 28
  19. 20233DiffTection: 3D Object Detection with Geometry-Aware Diffusion Featurescited by 26
  20. 2023CamoDiffusion: Camouflaged Object Detection via Conditional Diffusion Modelscited by 21
  21. 2023Diffusion Model for Dense Matchingcited by 21
  22. 2024SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flowcited by 20
  23. 2023DiffTAD: Temporal Action Detection with Proposal Denoising Diffusioncited by 19
  24. 2023DFormer: Diffusion-guided Transformer for Universal Image Segmentationcited by 19
  25. 2025PanSt3R: Multi-view Consistent Panoptic Segmentationcited by 19
  26. 2023SegGen: Supercharging Segmentation Models with Text2Mask and Mask2Img Synthesiscited by 18
  27. 2024A Simple Latent Diffusion Approach for Panoptic Segmentation and Mask Inpaintingcited by 18
  28. 2024Epsilon-VAE: Denoising as Visual Decodingcited by 16
  29. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real Worldcited by 16
  30. 2023Denoising Diffusion Semantic Segmentation with Mask Prior Modelingcited by 15
  31. 2024Scaling Properties of Diffusion Models for Perceptual Taskscited by 15
  32. 2023Diffusion-SS3D: Diffusion Model for Semi-supervised 3D Object Detectioncited by 14
  33. 2025From Image to Video: An Empirical Study of Diffusion Representationscited by 14
  34. 2023MonoDiffusion: Self-Supervised Monocular Depth Estimation Using Diffusion Modelcited by 12
  35. 2024Open-Vocabulary Segmentation with Unpaired Mask-Text Supervisioncited by 12
  36. 2024Explore In-Context Segmentation via Latent Diffusion Modelscited by 12
  37. 2023Diffusion for Natural Image Mattingcited by 11
  38. 2024IDOL: Unified Dual-Modal Latent Diffusion for Human-Centric Joint Video-Depth Generationcited by 11

50 most cited of 99 All 99 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025TRACE: Your Diffusion Model is Secretly an Instance Edge Detector
  2. 2020Towards Bounding-Box Free Panoptic Segmentation
  3. 2022Panoramic Panoptic Segmentation: Insights Into Surrounding Parsing for Mobile Agents via Unsupervised Contrastive Learning
  4. 2024SMITE: Segment Me In TimE
  5. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
  6. 2023MaskDiff: Modeling Mask Distribution with Diffusion Probabilistic Model for Few-Shot Instance Segmentation
  7. 2026MambaPanoptic: A Vision Mamba-based Structured State Space Framework for Panoptic Segmentation
  8. 2025Conditional Latent Diffusion Models for Zero-Shot Instance Segmentation