# SAM 2: Segment Anything in Images and Videos

arXiv: https://arxiv.org/abs/2408.00714 · HTML: https://mlsanity.com/paper/2408.00714/

## Key facts

- Cited by 2,863 papers in the map (data through 14 Oct 2026).
- Builds on 92 of its 126 references that are in the map.
- First posted on arXiv 1 Aug 2024.

## Builds on (most cited first, 30 of 92)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018)
- [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023)
- [Virtual KITTI 2](https://mlsanity.com/paper/2001.10773/) (2020)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles](https://mlsanity.com/paper/2306.00989/) (2023)
- [Rotary Position Embedding for Vision Transformer](https://mlsanity.com/paper/2403.13298/) (2024)
- [One-Shot Video Object Segmentation](https://mlsanity.com/paper/1611.05198/) (2016)
- [MOSE: A New Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2302.01872/) (2023)

## Built on by (most cited first, 30 of 2863)

- [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025)
- [Step1X-Edit: A Practical Framework for General Image Editing](https://mlsanity.com/paper/2504.17761/) (2025)
- [AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems](https://mlsanity.com/paper/2503.06669/) (2025)
- [Perception Encoder: The best visual embeddings are not at the output of the network](https://mlsanity.com/paper/2504.13181/) (2025)
- [GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation](https://mlsanity.com/paper/2410.06158/) (2024)
- [OmniGen2: Towards Instruction-Aligned Multimodal Generation](https://mlsanity.com/paper/2506.18871/) (2025)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement](https://mlsanity.com/paper/2503.06520/) (2025)
- [ImgEdit: A Unified Image Editing Dataset and Benchmark](https://mlsanity.com/paper/2505.20275/) (2025)
- [Video models are zero-shot learners and reasoners](https://mlsanity.com/paper/2509.20328/) (2025)
- [InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling](https://mlsanity.com/paper/2501.12386/) (2025)
- [World Simulation with Video Foundation Models for Physical AI](https://mlsanity.com/paper/2511.00062/) (2025)
- [EgoMimic: Scaling Imitation Learning via Egocentric Video](https://mlsanity.com/paper/2410.24221/) (2024)
- [FoundationStereo: Zero-Shot Stereo Matching](https://mlsanity.com/paper/2501.09898/) (2025)
- [Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos](https://mlsanity.com/paper/2501.04001/) (2025)
- [Identity-Preserving Text-to-Video Generation by Frequency Decomposition](https://mlsanity.com/paper/2411.17440/) (2024)
- [WorldScore: A Unified Evaluation Benchmark for World Generation](https://mlsanity.com/paper/2504.00983/) (2025)
- [RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics](https://mlsanity.com/paper/2411.16537/) (2024)
- [CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image](https://mlsanity.com/paper/2502.12894/) (2025)
- [Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control](https://mlsanity.com/paper/2503.14492/) (2025)
- [HD-EPIC: A Highly-Detailed Egocentric Video Dataset](https://mlsanity.com/paper/2502.04144/) (2025)
- [UniReal: Universal Image Generation and Editing via Learning Real-world Dynamics](https://mlsanity.com/paper/2412.07774/) (2024)
- [MedSAM2: Segment Anything in 3D Medical Images and Videos](https://mlsanity.com/paper/2504.03600/) (2025)
- [A Survey on Vision-Language-Action Models: An Action Tokenization Perspective](https://mlsanity.com/paper/2507.01925/) (2025)
- [PARTFIELD: Learning 3D Feature Fields for Part Segmentation and Beyond](https://mlsanity.com/paper/2504.11451/) (2025)
- [SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory](https://mlsanity.com/paper/2411.11922/) (2024)
- [Easi3R: Estimating Disentangled Motion from DUSt3R Without Training](https://mlsanity.com/paper/2503.24391/) (2025)
- [DINO-X: A Unified Vision Model for Open-World Object Detection and Understanding](https://mlsanity.com/paper/2411.14347/) (2024)
- [DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Grasping](https://mlsanity.com/paper/2502.20900/) (2025)
- [JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization](https://mlsanity.com/paper/2503.23377/) (2025)

## Abstract (arXiv)

We present Segment Anything Model 2 (SAM 2), a foundation model towards solving promptable visual segmentation in images and videos. We build a data engine, which improves model and data via user interaction, to collect the largest video segmentation dataset to date. Our model is a simple transformer architecture with streaming memory for real-time video processing. SAM 2 trained on our data provides strong performance across a wide range of tasks. In video segmentation, we observe better accuracy, using 3x fewer interactions than prior approaches. In image segmentation, our model is more accurate and 6x faster than the Segment Anything Model (SAM). We believe that our data, model, and insights will serve as a significant milestone for video segmentation and related perception tasks. We are releasing our main model, dataset, as well as code for model training and our demo.