# Segment Anything Meets Point Tracking

arXiv: https://arxiv.org/abs/2307.01197 · HTML: https://mlsanity.com/paper/2307.01197/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 70 references that are in the map.
- First posted on arXiv 3 Jul 2023.

## Builds on (most cited first, 30 of 48)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018)
- [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [GMFlow: Learning Optical Flow via Global Matching](https://mlsanity.com/paper/2111.13680/) (2021)
- [Images Speak in Images: A Generalist Painter for In-Context Visual Learning](https://mlsanity.com/paper/2212.02499/) (2022)
- [LIFT: Learned Invariant Feature Transform](https://mlsanity.com/paper/1603.09114/) (2016)
- [TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement](https://mlsanity.com/paper/2306.08637/) (2023)
- [Fast Online Object Tracking and Segmentation: A Unifying Approach](https://mlsanity.com/paper/1812.05050/) (2018)
- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [Space-Time Correspondence as a Contrastive Random Walk](https://mlsanity.com/paper/2006.14613/) (2020)
- [MOSE: A New Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2302.01872/) (2023)
- [Track Anything: Segment Anything Meets Videos](https://mlsanity.com/paper/2304.11968/) (2023)
- [Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories](https://mlsanity.com/paper/2204.04153/) (2022)
- [SegGPT: Segmenting Everything In Context](https://mlsanity.com/paper/2304.03284/) (2023)
- [Segment and Track Anything](https://mlsanity.com/paper/2305.06558/) (2023)
- [Tracking Everything Everywhere All at Once](https://mlsanity.com/paper/2306.05422/) (2023)
- [Mask2Former for Video Instance Segmentation](https://mlsanity.com/paper/2112.10764/) (2021)
- [Decoupling Features in Hierarchical Propagation for Video Object Segmentation](https://mlsanity.com/paper/2210.09782/) (2022)

## Built on by (most cited first, 30 of 81)

- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://mlsanity.com/paper/2310.11441/) (2023)
- [Multimodal Foundation Models: From Specialists to General-Purpose Assistants](https://mlsanity.com/paper/2309.10020/) (2023)
- [Tracking Anything with Decoupled Video Segmentation](https://mlsanity.com/paper/2309.03903/) (2023)
- [BootsTAP: Bootstrapped Training for Tracking-Any-Point](https://mlsanity.com/paper/2402.00847/) (2024)
- [Foundational Models Defining a New Era in Vision: A Survey and Outlook](https://mlsanity.com/paper/2307.13721/) (2023)
- [VideoVLA: Video Generators Can Be Generalizable Robot Manipulators](https://mlsanity.com/paper/2512.06963/) (2025)
- [Matching Anything by Segmenting Anything](https://mlsanity.com/paper/2406.04221/) (2024)
- [SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentation](https://mlsanity.com/paper/2311.17707/) (2023)
- [LVOS: A Benchmark for Large-scale Long-term Video Object Segmentation](https://mlsanity.com/paper/2404.19326/) (2024)
- [DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks](https://mlsanity.com/paper/2502.17157/) (2025)
- [Tracking and Mapping in Medical Computer Vision: A Review](https://mlsanity.com/paper/2310.11475/) (2023)
- [UniRef++: Segment Every Reference Object in Spatial and Temporal Spaces](https://mlsanity.com/paper/2312.15715/) (2023)
- [Efficient Track Anything](https://mlsanity.com/paper/2411.18933/) (2024)
- [SANeRF-HQ: Segment Anything for NeRF in High Quality](https://mlsanity.com/paper/2312.01531/) (2023)
- [Learning the What and How of Annotation in Video Object Segmentation](https://mlsanity.com/paper/2311.04414/) (2023)
- [PanoVOS: Bridging Non-panoramic and Panoramic Views with Transformer for Video Segmentation](https://mlsanity.com/paper/2309.12303/) (2023)
- [SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulation](https://mlsanity.com/paper/2405.19586/) (2024)
- [RepViT-SAM: Towards Real-Time Segmenting Anything](https://mlsanity.com/paper/2312.05760/) (2023)
- [LatentEditor: Text Driven Local Editing of 3D Scenes](https://mlsanity.com/paper/2312.09313/) (2023)
- [On Efficient Variants of Segment Anything Model: A Survey](https://mlsanity.com/paper/2410.04960/) (2024)
- [Can Visual Foundation Models Achieve Long-term Point Tracking?](https://mlsanity.com/paper/2408.13575/) (2024)
- [VideoMolmo: Spatio-Temporal Grounding Meets Pointing](https://mlsanity.com/paper/2506.05336/) (2025)
- [Zero-Shot Scene Change Detection](https://mlsanity.com/paper/2406.11210/) (2024)
- [SAM2 for Image and Video Segmentation: A Comprehensive Survey](https://mlsanity.com/paper/2503.12781/) (2025)
- [Generalizable Visual Reinforcement Learning with Segment Anything Model](https://mlsanity.com/paper/2312.17116/) (2023)
- [MAPLE: Encoding Dexterous Robotic Manipulation Priors Learned From Egocentric Videos](https://mlsanity.com/paper/2504.06084/) (2025)
- [3DEgo: 3D Editing on the Go!](https://mlsanity.com/paper/2407.10102/) (2024)
- [Caption Anything in Video: Fine-grained Object-centric Captioning via Spatiotemporal Multimodal Prompting](https://mlsanity.com/paper/2504.05541/) (2025)
- [Improving Visual Object Tracking through Visual Prompting](https://mlsanity.com/paper/2409.18901/) (2024)

## Abstract (arXiv)

The Segment Anything Model (SAM) has established itself as a powerful zero-shot image segmentation model, enabled by efficient point-centric annotation and prompt-based models. While click and brush interactions are both well explored in interactive image segmentation, the existing methods on videos focus on mask annotation and propagation. This paper presents SAM-PT, a novel method for point-centric interactive video segmentation, empowered by SAM and long-term point tracking. SAM-PT leverages robust and sparse point selection and propagation techniques for mask generation. Compared to traditional object-centric mask propagation strategies, we uniquely use point propagation to exploit local structure information agnostic to object semantics. We highlight the merits of point-based tracking through direct evaluation on the zero-shot open-world Unidentified Video Objects (UVO) benchmark. Our experiments on popular video object segmentation and multi-object segmentation tracking benchmarks, including DAVIS, YouTube-VOS, and BDD100K, suggest that a point-based segmentation tracker yields better zero-shot performance and efficient interactions. We release our code that integrates different point trackers and video segmentation benchmarks at https://github.com/SysCV/sam-pt.