# DiffusionInst: Diffusion Model for Instance Segmentation

arXiv: https://arxiv.org/abs/2212.02773 · HTML: https://mlsanity.com/paper/2212.02773/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 50 of its 60 references that are in the map.
- First posted on arXiv 6 Dec 2022.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 30 of 50)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [3D Shape Generation and Completion through Point-Voxel Diffusion](https://mlsanity.com/paper/2104.03670/) (2021)
- [Diffusion Models for Adversarial Purification](https://mlsanity.com/paper/2205.07460/) (2022)
- [DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models](https://mlsanity.com/paper/2210.08933/) (2022)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning](https://mlsanity.com/paper/2208.04202/) (2022)

## Built on by (most cited first, 30 of 49)

- [A Survey on Video Diffusion Models](https://mlsanity.com/paper/2310.10647/) (2023)
- [Dataset Diffusion: Diffusion-based Synthetic Dataset Generation for Pixel-Level Semantic Segmentation](https://mlsanity.com/paper/2309.14303/) (2023)
- [DiffusionTrack: Diffusion Model For Multi-Object Tracking](https://mlsanity.com/paper/2308.09905/) (2023)
- [DiffPose: SpatioTemporal Diffusion Model for Video-Based Human Pose Estimation](https://mlsanity.com/paper/2307.16687/) (2023)
- [DiffBEV: Conditional Diffusion Model for Bird's Eye View Perception](https://mlsanity.com/paper/2303.08333/) (2023)
- [SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process](https://mlsanity.com/paper/2312.12425/) (2023)
- [Diffusion Model for Camouflaged Object Detection](https://mlsanity.com/paper/2308.00303/) (2023)
- [Exploring Diffusion Models for Unsupervised Video Anomaly Detection](https://mlsanity.com/paper/2304.05841/) (2023)
- [Diffusion Model for Dense Matching](https://mlsanity.com/paper/2305.19094/) (2023)
- [CrowdDiff: Multi-hypothesis Crowd Density Estimation using Diffusion Models](https://mlsanity.com/paper/2303.12790/) (2023)
- [SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flow](https://mlsanity.com/paper/2405.20282/) (2024)
- [DFormer: Diffusion-guided Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2306.03437/) (2023)
- [A Simple Latent Diffusion Approach for Panoptic Segmentation and Mask Inpainting](https://mlsanity.com/paper/2401.10227/) (2024)
- [DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Data](https://mlsanity.com/paper/2403.15389/) (2024)
- [Denoising Diffusion Semantic Segmentation with Mask Prior Modeling](https://mlsanity.com/paper/2306.01721/) (2023)
- [Scaling Properties of Diffusion Models for Perceptual Tasks](https://mlsanity.com/paper/2411.08034/) (2024)
- [Diffusion-SS3D: Diffusion Model for Semi-supervised 3D Object Detection](https://mlsanity.com/paper/2312.02966/) (2023)
- [DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction](https://mlsanity.com/paper/2403.01226/) (2024)
- [Explore In-Context Segmentation via Latent Diffusion Models](https://mlsanity.com/paper/2403.09616/) (2024)
- [Diffusion for Natural Image Matting](https://mlsanity.com/paper/2312.05915/) (2023)
- [DiffUCD:Unsupervised Hyperspectral Image Change Detection with Semantic Correlation Diffusion Model](https://mlsanity.com/paper/2305.12410/) (2023)
- [Symmetrical Flow Matching: Unified Image Generation, Segmentation, and Classification with Score-Based Generative Models](https://mlsanity.com/paper/2506.10634/) (2025)
- [DiffSF: Diffusion Models for Scene Flow Estimation](https://mlsanity.com/paper/2403.05327/) (2024)
- [ResVG: Enhancing Relation and Semantic Understanding in Multiple Instances for Visual Grounding](https://mlsanity.com/paper/2408.16314/) (2024)
- [Learning Structure-Guided Diffusion Model for 2D Human Pose Estimation](https://mlsanity.com/paper/2306.17074/) (2023)
- [Fractal Calibration for long-tailed object detection](https://mlsanity.com/paper/2410.11774/) (2024)
- [SeeDiff: Off-the-Shelf Seeded Mask Generation from Diffusion Models](https://mlsanity.com/paper/2507.19808/) (2025)
- [Backpropagation Path Search On Adversarial Transferability](https://mlsanity.com/paper/2308.07625/) (2023)
- [3DifFusionDet: Diffusion Model for 3D Object Detection with Robust LiDAR-Camera Fusion](https://mlsanity.com/paper/2311.03742/) (2023)
- [GRIF-DM: Generation of Rich Impression Fonts using Diffusion Models](https://mlsanity.com/paper/2408.07259/) (2024)

## Abstract (arXiv)

Diffusion frameworks have achieved comparable performance with previous state-of-the-art image generation models. Researchers are curious about its variants in discriminative tasks because of its powerful noise-to-image denoising pipeline. This paper proposes DiffusionInst, a novel framework that represents instances as instance-aware filters and formulates instance segmentation as a noise-to-filter denoising process. The model is trained to reverse the noisy groundtruth without any inductive bias from RPN. During inference, it takes a randomly generated filter as input and outputs mask in one-step or multi-step denoising. Extensive experimental results on COCO and LVIS show that DiffusionInst achieves competitive performance compared to existing instance segmentation models with various backbones, such as ResNet and Swin Transformers. We hope our work could serve as a strong baseline, which could inspire designing more efficient diffusion frameworks for challenging discriminative tasks. Our code is available in https://github.com/chenhaoxing/DiffusionInst.