# Segment and Track Anything

arXiv: https://arxiv.org/abs/2305.06558 · HTML: https://mlsanity.com/paper/2305.06558/

## Key facts

- Cited by 193 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 23 references that are in the map.
- First posted on arXiv 11 May 2023.

## Builds on (most cited first, 17 of 17)

- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [Associating Objects with Transformers for Video Object Segmentation](https://mlsanity.com/paper/2106.02638/) (2021)
- [Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation](https://mlsanity.com/paper/2106.05210/) (2021)
- [Decoupling Features in Hierarchical Propagation for Video Object Segmentation](https://mlsanity.com/paper/2210.09782/) (2022)
- [Collaborative Video Object Segmentation by Foreground-Background Integration](https://mlsanity.com/paper/2003.08333/) (2020)
- [See More, Know More: Unsupervised Video Object Segmentation with Co-Attention Siamese Networks](https://mlsanity.com/paper/2001.06810/) (2020)
- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion](https://mlsanity.com/paper/2103.07941/) (2021)
- [Video Object Segmentation with Episodic Graph Memory Networks](https://mlsanity.com/paper/2007.07020/) (2020)
- [Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks](https://mlsanity.com/paper/2001.06807/) (2020)
- [Collaborative Video Object Segmentation by Multi-Scale Foreground-Background Integration](https://mlsanity.com/paper/2010.06349/) (2020)
- [Local-Global Context Aware Transformer for Language-Guided Video Segmentation](https://mlsanity.com/paper/2203.09773/) (2022)
- [Treating Motion as Option to Reduce Motion Dependency in Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2209.03138/) (2022)
- [Video Object Segmentation in Panoptic Wild Scenes](https://mlsanity.com/paper/2305.04470/) (2023)

## Built on by (most cited first, 30 of 193)

- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023)
- [EvalCrafter: Benchmarking and Evaluating Large Video Generation Models](https://mlsanity.com/paper/2310.11440/) (2023)
- [Tracking Anything with Decoupled Video Segmentation](https://mlsanity.com/paper/2309.03903/) (2023)
- [Putting the Object Back into Video Object Segmentation](https://mlsanity.com/paper/2310.12982/) (2023)
- [EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything](https://mlsanity.com/paper/2312.00863/) (2023)
- [ViPE: Video Pose Engine for 3D Geometric Perception](https://mlsanity.com/paper/2508.10934/) (2025)
- [L4GM: Large 4D Gaussian Reconstruction Model](https://mlsanity.com/paper/2406.10324/) (2024)
- [OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding](https://mlsanity.com/paper/2406.19389/) (2024)
- [Foundational Models Defining a New Era in Vision: A Survey and Outlook](https://mlsanity.com/paper/2307.13721/) (2023)
- [CoDeF: Content Deformation Fields for Temporally Consistent Video Processing](https://mlsanity.com/paper/2308.07926/) (2023)
- [Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Survey](https://mlsanity.com/paper/2307.04370/) (2023)
- [Segment Anything Meets Point Tracking](https://mlsanity.com/paper/2307.01197/) (2023)
- [Efficient Emotional Adaptation for Audio-Driven Talking-Head Generation](https://mlsanity.com/paper/2309.04946/) (2023)
- [HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video](https://mlsanity.com/paper/2311.18448/) (2023)
- [Review of Large Vision Models and Visual Prompt Engineering](https://mlsanity.com/paper/2307.00855/) (2023)
- [Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactively](https://mlsanity.com/paper/2401.02955/) (2024)
- [OMG-Seg: Is One Model Good Enough For All Segmentation?](https://mlsanity.com/paper/2401.10229/) (2024)
- [How NeRFs and 3D Gaussian Splatting are Reshaping SLAM: a Survey](https://mlsanity.com/paper/2402.13255/) (2024)
- [A Comprehensive Survey on Segment Anything Model for Vision and Beyond](https://mlsanity.com/paper/2305.08196/) (2023)
- [MagicStick: Controllable Video Editing via Control Handle Transformations](https://mlsanity.com/paper/2312.03047/) (2023)
- [DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)](https://mlsanity.com/paper/2401.08392/) (2024)
- [InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior](https://mlsanity.com/paper/2404.11613/) (2024)
- [CATR: Combinatorial-Dependence Audio-Queried Transformer for Audio-Visual Video Segmentation](https://mlsanity.com/paper/2309.09709/) (2023)
- [A Survey for Foundation Models in Autonomous Driving](https://mlsanity.com/paper/2402.01105/) (2024)
- [Follow-Your-Click: Open-domain Regional Image Animation via Short Prompts](https://mlsanity.com/paper/2403.08268/) (2024)
- [Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos](https://mlsanity.com/paper/2506.05302/) (2025)
- [Matching Anything by Segmenting Anything](https://mlsanity.com/paper/2406.04221/) (2024)
- [VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editing](https://mlsanity.com/paper/2502.17258/) (2025)
- [A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineering](https://mlsanity.com/paper/2306.06211/) (2023)

## Abstract (arXiv)

This report presents a framework called Segment And Track Anything (SAMTrack) that allows users to precisely and effectively segment and track any object in a video. Additionally, SAM-Track employs multimodal interaction methods that enable users to select multiple objects in videos for tracking, corresponding to their specific requirements. These interaction methods comprise click, stroke, and text, each possessing unique benefits and capable of being employed in combination. As a result, SAM-Track can be used across an array of fields, ranging from drone technology, autonomous driving, medical imaging, augmented reality, to biological analysis. SAM-Track amalgamates Segment Anything Model (SAM), an interactive key-frame segmentation model, with our proposed AOT-based tracking model (DeAOT), which secured 1st place in four tracks of the VOT 2022 challenge, to facilitate object tracking in video. In addition, SAM-Track incorporates Grounding-DINO, which enables the framework to support text-based interaction. We have demonstrated the remarkable capabilities of SAM-Track on DAVIS-2016 Val (92.0%), DAVIS-2017 Test (79.2%)and its practicability in diverse applications. The project page is available at: https://github.com/z-x-yang/Segment-and-Track-Anything.