# SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process

arXiv: https://arxiv.org/abs/2312.12425 · HTML: https://mlsanity.com/paper/2312.12425/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 55 of its 61 references that are in the map.
- First posted on arXiv 19 Dec 2023.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 30 of 55)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [Label-Efficient Semantic Segmentation with Diffusion Models](https://mlsanity.com/paper/2112.03126/) (2021)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning](https://mlsanity.com/paper/2208.04202/) (2022)
- [DiffusionDet: Diffusion Model for Object Detection](https://mlsanity.com/paper/2211.09788/) (2022)
- [Conditional Convolutions for Instance Segmentation](https://mlsanity.com/paper/2003.05664/) (2020)
- [SOLO: Segmenting Objects by Locations](https://mlsanity.com/paper/1912.04488/) (2019)
- [SegDiff: Image Segmentation with Diffusion Probabilistic Models](https://mlsanity.com/paper/2112.00390/) (2021)
- [Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise](https://mlsanity.com/paper/2208.09392/) (2022)

## Built on by (most cited first, 29 of 29)

- [Image Segmentation in Foundation Model Era: A Survey](https://mlsanity.com/paper/2408.12957/) (2024)
- [SAMRefiner: Taming Segment Anything Model for Universal Mask Refinement](https://mlsanity.com/paper/2502.06756/) (2025)
- [Digging into contrastive learning for robust depth estimation with diffusion models](https://mlsanity.com/paper/2404.09831/) (2024)
- [Mamba or RWKV: Exploring High-Quality and High-Efficiency Segment Anything Model](https://mlsanity.com/paper/2406.19369/) (2024)
- [[MASK] is All You Need](https://mlsanity.com/paper/2412.06787/) (2024)
- [Reversible Unfolding Network for Concealed Visual Perception with Generative Refinement](https://mlsanity.com/paper/2508.15027/) (2025)
- [PanoWorld-X: Generating Explorable Panoramic Worlds via Sphere-Aware Video Diffusion](https://mlsanity.com/paper/2509.24997/) (2025)
- [G4Seg: Generation for Inexact Segmentation Refinement with Diffusion Models](https://mlsanity.com/paper/2506.01539/) (2025)
- [ROSE: Revolutionizing Open-Set Dense Segmentation with Patch-Wise Perceptual Large Multimodal Model](https://mlsanity.com/paper/2412.00153/) (2024)
- [Uncertainty-Masked Bernoulli Diffusion for Camouflaged Object Detection Refinement](https://mlsanity.com/paper/2506.10712/) (2025)
- [Segment Any-Quality Images with Generative Latent Space Enhancement](https://mlsanity.com/paper/2503.12507/) (2025)
- [AnyLogo: Symbiotic Subject-Driven Diffusion System with Gemini Status](https://mlsanity.com/paper/2409.17740/) (2024)
- [rNCA: Self-Repairing Segmentation Masks](https://mlsanity.com/paper/2512.13397/) (2025)
- [Computationally Efficient Diffusion Models in Medical Imaging: A Comprehensive Review](https://mlsanity.com/paper/2505.07866/) (2025)
- [LawDIS: Language-Window-based Controllable Dichotomous Image Segmentation](https://mlsanity.com/paper/2508.01152/) (2025)
- [Learning from Adversity: Semantic-Aware Mask Refinement through Adversarial Perturbation](https://mlsanity.com/paper/2607.29059/) (2026)
- [Optimized Vessel Segmentation: A Structure-Agnostic Approach with Small Vessel Enhancement and Morphological Correction](https://mlsanity.com/paper/2411.15251/) (2024)
- [Objectomaly: Objectness-Aware Refinement for OoD Segmentation with Structural Consistency and Boundary Precision](https://mlsanity.com/paper/2507.07460/) (2025)
- [UGD-IML: A Unified Generative Diffusion-based Framework for Constrained and Unconstrained Image Manipulation Localization](https://mlsanity.com/paper/2508.06101/) (2025)
- [Towards Any-Quality Image Segmentation via Generative and Adaptive Latent Space Enhancement](https://mlsanity.com/paper/2601.02018/) (2026)
- [D3-RSMDE: 40× Faster and High-Fidelity Remote Sensing Monocular Depth Estimation](https://mlsanity.com/paper/2603.16362/) (2026)
- [Rethinking Vector Field Learning for Generative Segmentation](https://mlsanity.com/paper/2603.19218/) (2026)
- [Modular Diffusion Models for Structured Visual Recognition](https://mlsanity.com/paper/2606.22702/) (2026)
- [NeuroRefiner: Morphology-Aware Multi-Agent Refinement for 3D Fluorescence Microscopy Neuron Segmentation](https://mlsanity.com/paper/2608.09636/) (2026)
- [Generative Semantic Segmentation via an Observable Semantic-Image Interface and Hierarchical Generator Evidence Alignment](https://mlsanity.com/paper/2608.11537/) (2026)
- [Generative Semantic Scene Completion](https://mlsanity.com/paper/2608.26737/) (2026)
- [Evolving Error States: Failure-Aware Progressive Repair for Ultrasound Lesion Segmentation](https://mlsanity.com/paper/2609.18256/) (2026)
- [UnfoldCRF: Structured Mask Refinement with Image-Conditioned Latent Regions](https://mlsanity.com/paper/2609.33996/) (2026)
- [Controllable Road Marking Generation](https://mlsanity.com/paper/2610.05771/) (2026)

## Abstract (arXiv)

In this paper, we explore a principal way to enhance the quality of object masks produced by different segmentation models. We propose a model-agnostic solution called SegRefiner, which offers a novel perspective on this problem by interpreting segmentation refinement as a data generation process. As a result, the refinement process can be smoothly implemented through a series of denoising diffusion steps. Specifically, SegRefiner takes coarse masks as inputs and refines them using a discrete diffusion process. By predicting the label and corresponding states-transition probabilities for each pixel, SegRefiner progressively refines the noisy masks in a conditional denoising manner. To assess the effectiveness of SegRefiner, we conduct comprehensive experiments on various segmentation tasks, including semantic segmentation, instance segmentation, and dichotomous image segmentation. The results demonstrate the superiority of our SegRefiner from multiple aspects. Firstly, it consistently improves both the segmentation metrics and boundary metrics across different types of coarse masks. Secondly, it outperforms previous model-agnostic refinement methods by a significant margin. Lastly, it exhibits a strong capability to capture extremely fine details when refining high-resolution images. The source code and trained models are available at https://github.com/MengyuWang826/SegRefiner.