# Training-Free Robust Interactive Video Object Segmentation

arXiv: https://arxiv.org/abs/2406.05485 · HTML: https://mlsanity.com/paper/2406.05485/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 51 references that are in the map.
- First posted on arXiv 8 Jun 2024.

## Builds on (most cited first, 30 of 43)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018)
- [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023)
- [Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework](https://mlsanity.com/paper/2203.11991/) (2022)
- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [MOSE: A New Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2302.01872/) (2023)
- [Track Anything: Segment Anything Meets Videos](https://mlsanity.com/paper/2304.11968/) (2023)
- [Associating Objects with Transformers for Video Object Segmentation](https://mlsanity.com/paper/2106.02638/) (2021)
- [Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories](https://mlsanity.com/paper/2204.04153/) (2022)
- [Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation](https://mlsanity.com/paper/2106.05210/) (2021)
- [SegGPT: Segmenting Everything In Context](https://mlsanity.com/paper/2304.03284/) (2023)
- [Segment and Track Anything](https://mlsanity.com/paper/2305.06558/) (2023)
- [FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation](https://mlsanity.com/paper/1902.09513/) (2019)
- [Learning Video Object Segmentation from Static Images](https://mlsanity.com/paper/1612.02646/) (2016)
- [See More, Know More: Unsupervised Video Object Segmentation with Co-Attention Siamese Networks](https://mlsanity.com/paper/2001.06810/) (2020)
- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion](https://mlsanity.com/paper/2103.07941/) (2021)
- [Foundational Models Defining a New Era in Vision: A Survey and Outlook](https://mlsanity.com/paper/2307.13721/) (2023)
- [Video Object Segmentation with Adaptive Feature Bank and Uncertain-Region Refinement](https://mlsanity.com/paper/2010.07958/) (2020)
- [Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks](https://mlsanity.com/paper/2001.06807/) (2020)
- [Collaborative Video Object Segmentation by Multi-Scale Foreground-Background Integration](https://mlsanity.com/paper/2010.06349/) (2020)

## Built on by (most cited first, 2 of 2)

- [LaVin-DiT: Large Vision Diffusion Transformer](https://mlsanity.com/paper/2411.11505/) (2024)
- [AuralSAM2: Enabling SAM2 Hear Through Pyramid Audio-Visual Feature Prompting](https://mlsanity.com/paper/2506.01015/) (2025)

## Abstract (arXiv)

Interactive video object segmentation is a crucial video task, having various applications from video editing to data annotating. However, current approaches struggle to accurately segment objects across diverse domains. Recently, Segment Anything Model (SAM) introduces interactive visual prompts and demonstrates impressive performance across different domains. In this paper, we propose a training-free prompt tracking framework for interactive video object segmentation (I-PT), leveraging the powerful generalization of SAM. Although point tracking efficiently captures the pixel-wise information of objects in a video, points tend to be unstable when tracked over a long period, resulting in incorrect segmentation. Towards fast and robust interaction, we jointly adopt sparse points and boxes tracking, filtering out unstable points and capturing object-wise information. To better integrate reference information from multiple interactions, we introduce a cross-round space-time module (CRSTM), which adaptively aggregates mask features from previous rounds and frames, enhancing the segmentation stability. Our framework has demonstrated robust zero-shot video segmentation results on popular VOS datasets with interaction types, including DAVIS 2017, YouTube-VOS 2018, and MOSE 2023, maintaining a good tradeoff between performance and interaction time.