# SAM2MOT: A Novel Paradigm of Multi-Object Tracking by Segmentation

arXiv: https://arxiv.org/abs/2504.04519 · HTML: https://mlsanity.com/paper/2504.04519/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 55 references that are in the map.
- First posted on arXiv 6 Apr 2025.

## Builds on (most cited first, 30 of 43)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [TransTrack: Multiple Object Tracking with Transformer](https://mlsanity.com/paper/2012.15460/) (2020)
- [Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking](https://mlsanity.com/paper/2203.14360/) (2022)
- [DETRs with Collaborative Hybrid Assignments Training](https://mlsanity.com/paper/2211.12860/) (2022)
- [Track Anything: Segment Anything Meets Videos](https://mlsanity.com/paper/2304.11968/) (2023)
- [DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion](https://mlsanity.com/paper/2111.14690/) (2021)
- [The Unmanned Aerial Vehicle Benchmark: Object Detection and Tracking](https://mlsanity.com/paper/1804.00518/) (2018)
- [Universal Instance Perception as Object Discovery and Retrieval](https://mlsanity.com/paper/2303.06674/) (2023)
- [Track to Detect and Segment: An Online Multi-Object Tracker](https://mlsanity.com/paper/2103.08808/) (2021)
- [MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors](https://mlsanity.com/paper/2211.09791/) (2022)
- [MeMOT: Multi-Object Tracking with Memory](https://mlsanity.com/paper/2203.16761/) (2022)
- [Global Tracking Transformers](https://mlsanity.com/paper/2203.13250/) (2022)
- [SeqFormer: Sequential Transformer for Video Instance Segmentation](https://mlsanity.com/paper/2112.08275/) (2021)
- [SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory](https://mlsanity.com/paper/2411.11922/) (2024)
- [Towards Grand Unification of Object Tracking](https://mlsanity.com/paper/2207.07078/) (2022)
- [MeMOTR: Long-Term Memory-Augmented Transformer for Multi-Object Tracking](https://mlsanity.com/paper/2307.15700/) (2023)
- [SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Tree](https://mlsanity.com/paper/2410.16268/) (2024)
- [Hybrid-SORT: Weak Cues Matter for Online Multi-Object Tracking](https://mlsanity.com/paper/2308.00783/) (2023)

## Built on by (most cited first, 17 of 17)

- [Leader360V: The Large-scale, Real-world 360 Video Dataset for Multi-task Learning in Diverse Environment](https://mlsanity.com/paper/2506.14271/) (2025)
- [HiM2SAM: Enhancing SAM2 with Hierarchical Motion Estimation and Memory Optimization towards Long-term Tracking](https://mlsanity.com/paper/2507.07603/) (2025)
- [Seg2Track-SAM2: SAM2-based Multi-object Tracking and Segmentation](https://mlsanity.com/paper/2509.11772/) (2025)
- [GorillaWatch: An Automated System for In-the-Wild Gorilla Re-Identification and Population Monitoring](https://mlsanity.com/paper/2512.07776/) (2025)
- [HERO-VQL: Hierarchical, Egocentric and Robust Visual Query Localization](https://mlsanity.com/paper/2509.00385/) (2025)
- [AR-MOT: Autoregressive Multi-object Tracking](https://mlsanity.com/paper/2601.01925/) (2026)
- [SAM3-LiteText: An Anatomical Study of the SAM3 Text Encoder for Efficient Vision-Language Segmentation](https://mlsanity.com/paper/2602.12173/) (2026)
- [Hypergraph-State Collaborative Reasoning for Multi-Object Tracking](https://mlsanity.com/paper/2604.12665/) (2026)
- [ViewSAM: Learning View-aware Cross-modal Semantics for Weakly Supervised Cross-view Referring Multi-Object Tracking](https://mlsanity.com/paper/2605.02638/) (2026)
- [Tracking and Segmenting Anything in Any Modality](https://mlsanity.com/paper/2511.19475/) (2025)
- [SAMIDARE: Advanced Tracking-by-Segmentation for Dense Scenarios](https://mlsanity.com/paper/2604.22162/) (2026)
- [Zero-Shot Multi-Animal Tracking in the Wild](https://mlsanity.com/paper/2511.02591/) (2025)
- [Team-Aware Football Player Tracking with SAM: An Appearance-Based Approach to Occlusion Recovery](https://mlsanity.com/paper/2512.08467/) (2025)
- [SAMOFT: Robust Multi-Object Tracking via Region and Flow](https://mlsanity.com/paper/2605.09417/) (2026)
- [Seg2Track++: Probabilistic Track Validation and Data Association for Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/2606.03875/) (2026)
- [LiAM-SAM: Lifecycle-Aware Memory for Robust SAM2-Based MOT](https://mlsanity.com/paper/2609.28078/) (2026)
- [Towards Automatic Video Annotation with ASH: Zero-Shot Open-Vocabulary Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/2610.01022/) (2026)

## Abstract (arXiv)

Inspired by Segment Anything 2, which generalizes segmentation from images to videos, we propose SAM2MOT--a novel segmentation-driven paradigm for multi-object tracking that breaks away from the conventional detection-association framework. In contrast to previous approaches that treat segmentation as auxiliary information, SAM2MOT places it at the heart of the tracking process, systematically tackling challenges like false positives and occlusions. Its effectiveness has been thoroughly validated on major MOT benchmarks. Furthermore, SAM2MOT integrates pre-trained detector, pre-trained segmentor with tracking logic into a zero-shot MOT system that requires no fine-tuning. This significantly reduces dependence on labeled data and paves the way for transitioning MOT research from task-specific solutions to general-purpose systems. Experiments on DanceTrack, UAVDT, and BDD100K show state-of-the-art results. Notably, SAM2MOT outperforms existing methods on DanceTrack by +2.1 HOTA and +4.5 IDF1, highlighting its effectiveness in MOT. Code is available at https://github.com/TripleJoy/SAM2MOT.