mlsanityData to 7 Oct 2026

SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process

arXiv 2312.12425 ↗First posted 19 Dec 2023cs.CVcited by 28

On the map: Diffusion models → Medical imaging and segmentation

Abstract (arXiv)

In this paper, we explore a principal way to enhance the quality of object masks produced by different segmentation models. We propose a model-agnostic solution called SegRefiner, which offers a novel perspective on this problem by interpreting segmentation refinement as a data generation process. As a result, the refinement process can be smoothly implemented through a series of denoising diffusion steps. Specifically, SegRefiner takes coarse masks as inputs and refines them using a discrete diffusion process. By predicting the label and corresponding states-transition probabilities for each pixel, SegRefiner progressively refines the noisy masks in a conditional denoising manner. To assess the effectiveness of SegRefiner, we conduct comprehensive experiments on various segmentation tasks, including semantic segmentation, instance segmentation, and dichotomous image segmentation. The results demonstrate the superiority of our SegRefiner from multiple aspects. Firstly, it consistently improves both the segmentation metrics and boundary metrics across different types of coarse masks. Secondly, it outperforms previous model-agnostic refinement methods by a significant margin. Lastly, it exhibits a strong capability to capture extremely fine details when refining high-resolution images. The source code and trained models are available at https://github.com/MengyuWang826/SegRefiner.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. Builds on 55 of its 61 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

55 of 61 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformer19 Jan
  2. 2022DiffusionInst: Diffusion Model for Instance Segmentation6 Dec
  3. 2022MedSegDiff: Medical Image Segmentation with Diffusion Probabilistic Model1 Nov
  4. 2021Label-Efficient Semantic Segmentation with Diffusion Models6 Dec
  5. 2021SegDiff: Image Segmentation with Diffusion Probabilistic Models1 Dec

Other references · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2017Mask R-CNNcited by 7,259
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  10. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  12. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
38 more
  1. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  2. 2016Pyramid Scene Parsing Networkcited by 2,472
  3. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  4. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  5. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  6. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,417
  7. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  8. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  9. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 717
  10. 2016RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentationcited by 463
  11. 2019PointRend: Image Segmentation as Renderingcited by 439
  12. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  13. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  14. 2020Conditional Convolutions for Instance Segmentationcited by 314
  15. 2019SOLO: Segmenting Objects by Locationscited by 289
  16. 2022Cold Diffusion: Inverting Arbitrary Image Transforms Without Noisecited by 275
  17. 2017ICNet for Real-Time Semantic Segmentation on High-Resolution Imagescited by 261
  18. 2023MOSE: A New Dataset for Video Object Segmentation in Complex Scenescited by 232
  19. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 225
  20. 2021Boundary IoU: Improving Object-Centric Image Segmentation Evaluationcited by 186
  21. 2021Instances as Queriescited by 157
  22. 2021Camouflaged Object Segmentation with Distraction Miningcited by 151
  23. 2023DDP: Diffusion Model for Dense Visual Predictioncited by 148
  24. 2020CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinementcited by 113
  25. 2022Highly Accurate Dichotomous Image Segmentationcited by 110
  26. 2022A Generalist Framework for Panoptic Segmentation of Images and Videoscited by 99
  27. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesiscited by 99
  28. 2021SeqFormer: Sequential Transformer for Video Instance Segmentationcited by 98
  29. 2020SegFix: Model-Agnostic Boundary Refinement for Segmentationcited by 97
  30. 2019PolyTransform: Deep Polygon Transformer for Instance Segmentationcited by 92
  31. 2020Mask Guided Matting via Progressive Refinement Networkcited by 84
  32. 2021Mask Transfiner for High-Quality Instance Segmentationcited by 64
  33. 2020AlignSeg: Feature-Aligned Segmentation Networkscited by 53
  34. 2021RefineMask: Towards High-Quality Instance Segmentation with Fine-Grained Featurescited by 47
  35. 2021Look Closer to Segment Better: Boundary Patch Refinement for Instance Segmentationcited by 43
  36. 2021High Quality Segmentation for Ultra High-resolution Imagescited by 31
  37. 2022Video Mask Transfiner for High-Quality Video Instance Segmentationcited by 28
  38. 2020DeepStrip: High Resolution Boundary Refinementcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 11; 2026: 10.

28 papers.

Sort:

By area:

  1. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  2. 2024Digging into contrastive learning for robust depth estimation with diffusion modelscited by 19
  3. 2025SAMRefiner: Taming Segment Anything Model for Universal Mask Refinementcited by 19
  4. 2024Mamba or RWKV: Exploring High-Quality and High-Efficiency Segment Anything Modelcited by 18
  5. 2024[MASK] is All You Needcited by 11
  6. 2025Reversible Unfolding Network for Concealed Visual Perception with Generative Refinementcited by 11
  7. 2025PanoWorld-X: Generating Explorable Panoramic Worlds via Sphere-Aware Video Diffusioncited by 7
  8. 2025G4Seg: Generation for Inexact Segmentation Refinement with Diffusion Modelscited by 5
  9. 2024ROSE: Revolutionizing Open-Set Dense Segmentation with Patch-Wise Perceptual Large Multimodal Modelcited by 4
  10. 2025Uncertainty-Masked Bernoulli Diffusion for Camouflaged Object Detection Refinementcited by 4
  11. 2025Segment Any-Quality Images with Generative Latent Space Enhancementcited by 3
  12. 2024AnyLogo: Symbiotic Subject-Driven Diffusion System with Gemini Statuscited by 2
16 more
  1. 2025rNCA: Self-Repairing Segmentation Maskscited by 2
  2. 2025Computationally Efficient Diffusion Models in Medical Imaging: A Comprehensive Reviewcited by 1
  3. 2025LawDIS: Language-Window-based Controllable Dichotomous Image Segmentationcited by 1
  4. 2026Learning from Adversity: Semantic-Aware Mask Refinement through Adversarial Perturbationcited by 1
  5. 2024Optimized Vessel Segmentation: A Structure-Agnostic Approach with Small Vessel Enhancement and Morphological Correctioncited by 0
  6. 2025Objectomaly: Objectness-Aware Refinement for OoD Segmentation with Structural Consistency and Boundary Precisioncited by 0
  7. 2025UGD-IML: A Unified Generative Diffusion-based Framework for Constrained and Unconstrained Image Manipulation Localizationcited by 0
  8. 2026Towards Any-Quality Image Segmentation via Generative and Adaptive Latent Space Enhancementcited by 0
  9. 2026D3-RSMDE: 40× Faster and High-Fidelity Remote Sensing Monocular Depth Estimationcited by 0
  10. 2026Rethinking Vector Field Learning for Generative Segmentationcited by 0
  11. 2026Modular Diffusion Models for Structured Visual Recognitioncited by 0
  12. 2026NeuroRefiner: Morphology-Aware Multi-Agent Refinement for 3D Fluorescence Microscopy Neuron Segmentationcited by 0
  13. 2026Generative Semantic Segmentation via an Observable Semantic-Image Interface and Hierarchical Generator Evidence Alignmentcited by 0
  14. 2026Generative Semantic Scene Completioncited by 0
  15. 2026Evolving Error States: Failure-Aware Progressive Repair for Ultrasound Lesion Segmentationcited by 0
  16. 2026UnfoldCRF: Structured Mask Refinement with Image-Conditioned Latent Regionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Denoising Diffusion Semantic Segmentation with Mask Prior Modeling
  2. 2023DFormer: Diffusion-guided Transformer for Universal Image Segmentation
  3. 2024Hybrid diffusion models: combining supervised and generative pretraining for label-efficient fine-tuning of segmentation models
  4. 2024Free-Mask: A Novel Paradigm of Integration Between the Segmentation Diffusion Model and Image Editing
  5. 2024Mask Factory: Towards High-quality Synthetic Data Generation for Dichotomous Image Segmentation
  6. 2026Towards Agnostic and Holistic Universal Image Segmentation with Bit Diffusion
  7. 2023Factorized Diffusion Architectures for Unsupervised Image Generation and Segmentation
  8. 2026Contour Refinement using Discrete Diffusion in Low Data Regime