# Reviving Iterative Training with Mask Guidance for Interactive Segmentation

arXiv: https://arxiv.org/abs/2102.06583 · HTML: https://mlsanity.com/paper/2102.06583/

## Key facts

- Cited by 163 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 51 references that are in the map.
- First posted on arXiv 12 Feb 2021.

## Builds on (most cited first, 24 of 24)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [Balanced Meta-Softmax for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2007.10740/) (2020)
- [ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1604.05144/) (2016)
- [Equalization Loss for Long-Tailed Object Recognition](https://mlsanity.com/paper/2003.05176/) (2020)
- [Deep Interactive Object Selection](https://mlsanity.com/paper/1603.04042/) (2016)
- [Deep Extreme Cut: From Extreme Points to Object Segmentation](https://mlsanity.com/paper/1711.09081/) (2017)
- [Efficient Interactive Annotation of Segmentation Datasets with Polygon-RNN++](https://mlsanity.com/paper/1803.09693/) (2018)
- [f-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation](https://mlsanity.com/paper/2001.10331/) (2020)
- [Large-scale interactive object segmentation with human annotators](https://mlsanity.com/paper/1903.10830/) (2019)
- [SegFix: Model-Agnostic Boundary Refinement for Segmentation](https://mlsanity.com/paper/2007.04269/) (2020)
- [Learning to Refine Object Segments](https://mlsanity.com/paper/1603.08695/) (2016)
- [AdaptIS: Adaptive Instance Selection Network](https://mlsanity.com/paper/1909.07829/) (2019)
- [Learning to Segment the Tail](https://mlsanity.com/paper/2004.00900/) (2020)
- [Iteratively Trained Interactive Segmentation](https://mlsanity.com/paper/1805.04398/) (2018)
- [Continuous Adaptation for Interactive Object Segmentation by Learning from Corrections](https://mlsanity.com/paper/1911.12709/) (2019)
- [Interactive Full Image Segmentation by Considering All Regions Jointly](https://mlsanity.com/paper/1812.01888/) (2018)
- [DeepStrip: High Resolution Boundary Refinement](https://mlsanity.com/paper/2003.11670/) (2020)

## Built on by (most cited first, 30 of 163)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [Putting the Object Back into Video Object Segmentation](https://mlsanity.com/paper/2310.12982/) (2023)
- [Segment Any 3D Gaussians](https://mlsanity.com/paper/2312.00860/) (2023)
- [SimpleClick: Interactive Image Segmentation with Simple Vision Transformers](https://mlsanity.com/paper/2210.11006/) (2022)
- [FocalClick: Towards Practical Interactive Image Segmentation](https://mlsanity.com/paper/2204.02574/) (2022)
- ["This is my unicorn, Fluffy": Personalizing frozen vision-language representations](https://mlsanity.com/paper/2204.01694/) (2022)
- [OmniSeg3D: Omniversal 3D Segmentation via Hierarchical Contrastive Learning](https://mlsanity.com/paper/2311.11666/) (2023)
- [General Object Foundation Model for Images and Videos at Scale](https://mlsanity.com/paper/2312.09158/) (2023)
- [SAM 3D Body: Robust Full-Body Human Mesh Recovery](https://mlsanity.com/paper/2602.15989/) (2026)
- [Segment Anything in 3D with Radiance Fields](https://mlsanity.com/paper/2304.12308/) (2023)
- [ScribblePrompt: Fast and Flexible Interactive Segmentation for Any Biomedical Image](https://mlsanity.com/paper/2312.07381/) (2023)
- [PseudoClick: Interactive Image Segmentation with Click Imitation](https://mlsanity.com/paper/2207.05282/) (2022)
- [Segment anything, from space?](https://mlsanity.com/paper/2304.13000/) (2023)
- [EdgeSAM: Prompt-In-the-Loop Distillation for SAM](https://mlsanity.com/paper/2312.06660/) (2023)
- [EdgeFlow: Achieving Practical Interactive Segmentation with Edge-Guided Flow](https://mlsanity.com/paper/2109.09406/) (2021)
- [AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation](https://mlsanity.com/paper/2306.00977/) (2023)
- [Interactive Medical Image Segmentation: A Benchmark Dataset and Baseline](https://mlsanity.com/paper/2411.12814/) (2024)
- [SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories](https://mlsanity.com/paper/2503.08625/) (2025)
- [iSegFormer: Interactive Segmentation via Transformers with Application to 3D Knee MR Images](https://mlsanity.com/paper/2112.11325/) (2021)
- [Semantic Segmentation In-the-Wild Without Seeing Any Segmentation Examples](https://mlsanity.com/paper/2112.03185/) (2021)
- [What a MESS: Multi-Domain Evaluation of Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2306.15521/) (2023)
- [Interactive Object Segmentation in 3D Point Clouds](https://mlsanity.com/paper/2204.07183/) (2022)
- [LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision Foundation Models](https://mlsanity.com/paper/2504.14032/) (2025)
- [Rethinking Interactive Image Segmentation with Low Latency, High Quality, and Diverse Prompts](https://mlsanity.com/paper/2404.00741/) (2024)
- [Pointly-Supervised Panoptic Segmentation](https://mlsanity.com/paper/2210.13950/) (2022)
- [InterFormer: Real-time Interactive Image Segmentation](https://mlsanity.com/paper/2304.02942/) (2023)

## Abstract (arXiv)

Recent works on click-based interactive segmentation have demonstrated state-of-the-art results by using various inference-time optimization schemes. These methods are considerably more computationally expensive compared to feedforward approaches, as they require performing backward passes through a network during inference and are hard to deploy on mobile frameworks that usually support only forward passes. In this paper, we extensively evaluate various design choices for interactive segmentation and discover that new state-of-the-art results can be obtained without any additional optimization schemes. Thus, we propose a simple feedforward model for click-based interactive segmentation that employs the segmentation masks from previous steps. It allows not only to segment an entirely new object, but also to start with an external mask and correct it. When analyzing the performance of models trained on different datasets, we observe that the choice of a training dataset greatly impacts the quality of interactive segmentation. We find that the models trained on a combination of COCO and LVIS with diverse and high-quality annotations show performance superior to all existing models. The code and trained models are available at https://github.com/saic-vul/ritm_interactive_segmentation.