# SAMRefiner: Taming Segment Anything Model for Universal Mask Refinement

arXiv: https://arxiv.org/abs/2502.06756 · HTML: https://mlsanity.com/paper/2502.06756/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 66 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Builds on (most cited first, 30 of 55)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023)
- [ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1604.05144/) (2016)
- [Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/1803.10464/) (2018)
- [Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2004.04581/) (2020)
- [Personalize Segment Anything Model with One Shot](https://mlsanity.com/paper/2305.03048/) (2023)
- [BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1503.01640/) (2015)
- [Track Anything: Segment Anything Meets Videos](https://mlsanity.com/paper/2304.11968/) (2023)
- [CANet: Class-Agnostic Segmentation Networks with Iterative Refinement and Attentive Few-Shot Learning](https://mlsanity.com/paper/1903.02351/) (2019)
- [Segment and Track Anything](https://mlsanity.com/paper/2305.06558/) (2023)
- [Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals](https://mlsanity.com/paper/2102.06191/) (2021)
- [Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentation](https://mlsanity.com/paper/1502.02734/) (2015)
- [Boundary IoU: Improving Object-Centric Image Segmentation Evaluation](https://mlsanity.com/paper/2103.16562/) (2021)
- [Cut and Learn for Unsupervised Object Detection and Instance Segmentation](https://mlsanity.com/paper/2301.11320/) (2023)
- [ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation](https://mlsanity.com/paper/2106.05095/) (2021)
- [Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2208.09910/) (2022)
- [PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering](https://mlsanity.com/paper/2103.17070/) (2021)
- [Deep Image Matting](https://mlsanity.com/paper/1703.03872/) (2017)

## Built on by (most cited first, 20 of 20)

- [Reversible Unfolding Network for Concealed Visual Perception with Generative Refinement](https://mlsanity.com/paper/2508.15027/) (2025)
- [Text4Seg++: Advancing Image Segmentation via Generative Language Modeling](https://mlsanity.com/paper/2509.06321/) (2025)
- [RADSeg: Unleashing Parameter and Compute Efficient Zero-Shot Open-Vocabulary Segmentation Using Agglomerative Models](https://mlsanity.com/paper/2511.19704/) (2025)
- [Uncertainty-Masked Bernoulli Diffusion for Camouflaged Object Detection Refinement](https://mlsanity.com/paper/2506.10712/) (2025)
- [Prompt Engineering in Segment Anything Model: Methodologies, Applications, and Emerging Challenges](https://mlsanity.com/paper/2507.09562/) (2025)
- [OVerSeeC: Open-Vocabulary Costmap Generation from Satellite Images and Natural Language](https://mlsanity.com/paper/2602.18606/) (2026)
- [A drone-based framework for coral habitat mapping via weakly supervised segmentation](https://mlsanity.com/paper/2508.18958/) (2025)
- [SAM 2++: Tracking Anything at Any Granularity](https://mlsanity.com/paper/2510.18822/) (2025)
- [Remote SAMsing: From Segment Anything to Segment Everything](https://mlsanity.com/paper/2605.00256/) (2026)
- [Learning from Adversity: Semantic-Aware Mask Refinement through Adversarial Perturbation](https://mlsanity.com/paper/2607.29059/) (2026)
- [Domain and Task-Focused Example Selection for Data-Efficient Contrastive Medical Image Segmentation](https://mlsanity.com/paper/2505.19208/) (2025)
- [Segment and Matte Anything in a Unified Model](https://mlsanity.com/paper/2601.12147/) (2026)
- [Learning from Noisy Prompts: Saliency-Guided Prompt Distillation for Robust Segmentation with SAM](https://mlsanity.com/paper/2604.23314/) (2026)
- [Dual-Foundation Models for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2605.03365/) (2026)
- [DGSeg: Dynamic Gating of Semantic-Spatial Guided Predictions for Reasoning Segmentation](https://mlsanity.com/paper/2607.04779/) (2026)
- [Toward Mask Annotation-Free Surgical Instrument Segmentation from Endoscopic Images Using Text-Prompted Segment Anything Model 3 (SAM3)](https://mlsanity.com/paper/2608.08844/) (2026)
- [Leveraging existing sparse point annotations for benthic imagery dense segmentation](https://mlsanity.com/paper/2608.17561/) (2026)
- [Can Frozen Hyperspherical Features Guide the Selection of Pseudo Masks?](https://mlsanity.com/paper/2609.30080/) (2026)
- [UnfoldCRF: Structured Mask Refinement with Image-Conditioned Latent Regions](https://mlsanity.com/paper/2609.33996/) (2026)
- [Segment Where You Look: Attention as the Interface Between Language Models and Segmenters](https://mlsanity.com/paper/2610.18729/) (2026)

## Abstract (arXiv)

In this paper, we explore a principal way to enhance the quality of widely pre-existing coarse masks, enabling them to serve as reliable training data for segmentation models to reduce the annotation cost. In contrast to prior refinement techniques that are tailored to specific models or tasks in a close-world manner, we propose SAMRefiner, a universal and efficient approach by adapting SAM to the mask refinement task. The core technique of our model is the noise-tolerant prompting scheme. Specifically, we introduce a multi-prompt excavation strategy to mine diverse input prompts for SAM (i.e., distance-guided points, context-aware elastic bounding boxes, and Gaussian-style masks) from initial coarse masks. These prompts can collaborate with each other to mitigate the effect of defects in coarse masks. In particular, considering the difficulty of SAM to handle the multi-object case in semantic segmentation, we introduce a split-then-merge (STM) pipeline. Additionally, we extend our method to SAMRefiner++ by introducing an additional IoU adaption step to further boost the performance of the generic SAMRefiner on the target dataset. This step is self-boosted and requires no additional annotation. The proposed framework is versatile and can flexibly cooperate with existing segmentation methods. We evaluate our mask framework on a wide range of benchmarks under different settings, demonstrating better accuracy and efficiency. SAMRefiner holds significant potential to expedite the evolution of refinement tools. Our code is available at https://github.com/linyq2117/SAMRefiner.