# EdgeTAM: On-Device Track Anything Model

arXiv: https://arxiv.org/abs/2501.07256 · HTML: https://mlsanity.com/paper/2501.07256/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 70 of its 84 references that are in the map.
- First posted on arXiv 13 Jan 2025.

## Builds on (most cited first, 30 of 70)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations](https://mlsanity.com/paper/1707.03237/) (2017)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles](https://mlsanity.com/paper/2306.00989/) (2023)
- [Fast Segment Anything](https://mlsanity.com/paper/2306.12156/) (2023)
- [One-Shot Video Object Segmentation](https://mlsanity.com/paper/1611.05198/) (2016)
- [MOSE: A New Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2302.01872/) (2023)
- [YouTube-VOS: Sequence-to-Sequence Video Object Segmentation](https://mlsanity.com/paper/1809.00461/) (2018)
- [Segment Anything in Medical Images](https://mlsanity.com/paper/2304.12306/) (2023)
- [Associating Objects with Transformers for Video Object Segmentation](https://mlsanity.com/paper/2106.02638/) (2021)
- [Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation](https://mlsanity.com/paper/2106.05210/) (2021)
- [Putting the Object Back into Video Object Segmentation](https://mlsanity.com/paper/2310.12982/) (2023)
- [EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything](https://mlsanity.com/paper/2312.00863/) (2023)
- [RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Model](https://mlsanity.com/paper/2306.16269/) (2023)
- [FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation](https://mlsanity.com/paper/1902.09513/) (2019)

## Built on by (most cited first, 21 of 21)

- [SAM2MOT: A Novel Paradigm of Multi-Object Tracking by Segmentation](https://mlsanity.com/paper/2504.04519/) (2025)
- [RAM: Recover Any 3D Human Motion in-the-Wild](https://mlsanity.com/paper/2603.19929/) (2026)
- [Leader360V: The Large-scale, Real-world 360 Video Dataset for Multi-task Learning in Diverse Environment](https://mlsanity.com/paper/2506.14271/) (2025)
- [EfficientSAM3: Progressive Hierarchical Distillation for Video Concept Segmentation from SAM1, 2, and 3](https://mlsanity.com/paper/2511.15833/) (2025)
- [3AM: 3egment Anything with Geometric Consistency in Videos](https://mlsanity.com/paper/2601.08831/) (2026)
- [Fast SAM2 with Text-Driven Token Pruning](https://mlsanity.com/paper/2512.21333/) (2025)
- [Segment Anything for Video: A Comprehensive Review of Video Object Segmentation and Tracking from Past to Future](https://mlsanity.com/paper/2507.22792/) (2025)
- [Distractor-Aware Memory-Based Visual Object Tracking](https://mlsanity.com/paper/2509.13864/) (2025)
- [Efficient-SAM2: Accelerating SAM2 with Object-Aware Visual Encoding and Memory Retrieval](https://mlsanity.com/paper/2602.08224/) (2026)
- [SAM 2++: Tracking Anything at Any Granularity](https://mlsanity.com/paper/2510.18822/) (2025)
- [RobotSeg: A Model and Dataset for Segmenting Robots in Image and Video](https://mlsanity.com/paper/2511.22950/) (2025)
- [Colon-Bench: An Agentic Workflow for Scalable Dense Lesion Annotation in Full-Procedure Colonoscopy Videos](https://mlsanity.com/paper/2603.25645/) (2026)
- [Backdoor Attacks on Prompt-Driven Video Segmentation Foundation Models](https://mlsanity.com/paper/2512.22046/) (2025)
- [Scaling Ultrasound Volumetric Reconstruction via Mobile Augmented Reality](https://mlsanity.com/paper/2602.18500/) (2026)
- [EdgeDAM: Real-time Object Tracking for Mobile Devices](https://mlsanity.com/paper/2603.05463/) (2026)
- [REST: Receding Horizon Explorative Steiner Tree for Zero-Shot Object-Goal Navigation](https://mlsanity.com/paper/2603.18624/) (2026)
- [Efficient Tracking and Understanding Object Transformations](https://mlsanity.com/paper/2607.19743/) (2026)
- [Lean-SAM2: Target-Anchored Memory and Encoder Acceleration for SAM2](https://mlsanity.com/paper/2607.19811/) (2026)
- [StreamDAM: Presence-Aware Memory for Real-Time Streaming Video Object Segmentation](https://mlsanity.com/paper/2608.03912/) (2026)
- [Cloak of Invisibility: Real-Time Privacy-Preserving Volumetric Video Streaming](https://mlsanity.com/paper/2608.11645/) (2026)
- [Training-Free Long-Term Multi-Object Tracking for Sports Video Analytics](https://mlsanity.com/paper/2608.15688/) (2026)

## Abstract (arXiv)

On top of Segment Anything Model (SAM), SAM 2 further extends its capability from image to video inputs through a memory bank mechanism and obtains a remarkable performance compared with previous methods, making it a foundation model for video segmentation task. In this paper, we aim at making SAM 2 much more efficient so that it even runs on mobile devices while maintaining a comparable performance. Despite several works optimizing SAM for better efficiency, we find they are not sufficient for SAM 2 because they all focus on compressing the image encoder, while our benchmark shows that the newly introduced memory attention blocks are also the latency bottleneck. Given this observation, we propose EdgeTAM, which leverages a novel 2D Spatial Perceiver to reduce the computational cost. In particular, the proposed 2D Spatial Perceiver encodes the densely stored frame-level memories with a lightweight Transformer that contains a fixed set of learnable queries. Given that video segmentation is a dense prediction task, we find preserving the spatial structure of the memories is essential so that the queries are split into global-level and patch-level groups. We also propose a distillation pipeline that further improves the performance without inference overhead. As a result, EdgeTAM achieves 87.7, 70.0, 72.3, and 71.7 J&F on DAVIS 2017, MOSE, SA-V val, and SA-V test, while running at 16 FPS on iPhone 15 Pro Max.