# A Comprehensive Survey on Segment Anything Model for Vision and Beyond

arXiv: https://arxiv.org/abs/2305.08196 · HTML: https://mlsanity.com/paper/2305.08196/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Builds on 168 of its 223 references that are in the map.
- First posted on arXiv 14 May 2023.

## Builds on (most cited first, 30 of 168)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)

## Built on by (most cited first, 30 of 64)

- [Multimodal Foundation Models: From Specialists to General-Purpose Assistants](https://mlsanity.com/paper/2309.10020/) (2023)
- [Foundational Models Defining a New Era in Vision: A Survey and Outlook](https://mlsanity.com/paper/2307.13721/) (2023)
- [Review of Large Vision Models and Visual Prompt Engineering](https://mlsanity.com/paper/2307.00855/) (2023)
- [ClipSAM: CLIP and SAM Collaboration for Zero-Shot Anomaly Segmentation](https://mlsanity.com/paper/2401.12665/) (2024)
- [SAMAug: Point Prompt Augmentation for Segment Anything Model](https://mlsanity.com/paper/2307.01187/) (2023)
- [Image Segmentation in Foundation Model Era: A Survey](https://mlsanity.com/paper/2408.12957/) (2024)
- [GoodSAM: Bridging Domain and Capacity Gaps via Segment Anything Model for Distortion-aware Panoramic Semantic Segmentation](https://mlsanity.com/paper/2403.16370/) (2024)
- [Detect Any Deepfakes: Segment Anything Meets Face Forgery Detection and Localization](https://mlsanity.com/paper/2306.17075/) (2023)
- [AutoProSAM: Automated Prompting SAM for 3D Multi-Organ Segmentation](https://mlsanity.com/paper/2308.14936/) (2023)
- [Interactive 3D Medical Image Segmentation with SAM 2](https://mlsanity.com/paper/2408.02635/) (2024)
- [On Efficient Variants of Segment Anything Model: A Survey](https://mlsanity.com/paper/2410.04960/) (2024)
- [Segment Anything Model is a Good Teacher for Local Feature Learning](https://mlsanity.com/paper/2309.16992/) (2023)
- [WebUOT-1M: Advancing Deep Underwater Object Tracking with A Million-Scale Benchmark](https://mlsanity.com/paper/2405.19818/) (2024)
- [SAD: Segment Any RGBD](https://mlsanity.com/paper/2305.14207/) (2023)
- [SAM-OCTA: A Fine-Tuning Strategy for Applying Foundation Model to OCTA Image Segmentation Tasks](https://mlsanity.com/paper/2309.11758/) (2023)
- [Tree-GPT: Modular Large Language Model Expert System for Forest Remote Sensing Image Understanding and Interactive Analysis](https://mlsanity.com/paper/2310.04698/) (2023)
- [See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition](https://mlsanity.com/paper/2407.05417/) (2024)
- [SAM2 for Image and Video Segmentation: A Comprehensive Survey](https://mlsanity.com/paper/2503.12781/) (2025)
- [GoodSAM++: Bridging Domain and Capacity Gaps via Segment Anything Model for Panoramic Semantic Segmentation](https://mlsanity.com/paper/2408.09115/) (2024)
- [Diffusion-empowered AutoPrompt MedSAM](https://mlsanity.com/paper/2502.06817/) (2025)
- [Temporally-Extended Prompts Optimization for SAM in Interactive Medical Image Segmentation](https://mlsanity.com/paper/2306.08958/) (2023)
- [WeakMedSAM: Weakly-Supervised Medical Image Segmentation via SAM with Sub-Class Exploration and Prompt Affinity Mining](https://mlsanity.com/paper/2503.04106/) (2025)
- [Large Scale Foundation Models for Intelligent Manufacturing Applications: A Survey](https://mlsanity.com/paper/2312.06718/) (2023)
- [SAM-OCTA: Prompting Segment-Anything for OCTA Image Segmentation](https://mlsanity.com/paper/2310.07183/) (2023)
- [Segment Anything for Videos: A Systematic Survey](https://mlsanity.com/paper/2408.08315/) (2024)
- [Underwater Camouflaged Object Tracking Meets Vision-Language SAM2](https://mlsanity.com/paper/2409.16902/) (2024)
- [Real-Time Localization and Bimodal Point Pattern Analysis of Palms Using UAV Imagery](https://mlsanity.com/paper/2410.11124/) (2024)
- [MambaTrack: Exploiting Dual-Enhancement for Night UAV Tracking](https://mlsanity.com/paper/2411.15761/) (2024)
- [Industrial Segment Anything -- a Case Study in Aircraft Manufacturing, Intralogistics, Maintenance, Repair, and Overhaul](https://mlsanity.com/paper/2307.12674/) (2023)
- [Promoting Segment Anything Model towards Highly Accurate Dichotomous Image Segmentation](https://mlsanity.com/paper/2401.00248/) (2023)

## Abstract (arXiv)

Artificial intelligence (AI) is evolving towards artificial general intelligence, which refers to the ability of an AI system to perform a wide range of tasks and exhibit a level of intelligence similar to that of a human being. This is in contrast to narrow or specialized AI, which is designed to perform specific tasks with a high degree of efficiency. Therefore, it is urgent to design a general class of models, which we term foundation models, trained on broad data that can be adapted to various downstream tasks. The recently proposed segment anything model (SAM) has made significant progress in breaking the boundaries of segmentation, greatly promoting the development of foundation models for computer vision. To fully comprehend SAM, we conduct a survey study. As the first to comprehensively review the progress of segmenting anything task for vision and beyond based on the foundation model of SAM, this work focuses on its applications to various tasks and data types by discussing its historical development, recent progress, and profound impact on broad applications. We first introduce the background and terminology for foundation models including SAM, as well as state-of-the-art methods contemporaneous with SAM that are significant for segmenting anything task. Then, we analyze and summarize the advantages and limitations of SAM across various image processing applications, including software scenes, real-world scenes, and complex scenes. Importantly, many insights are drawn to guide future research to develop more versatile foundation models and improve the architecture of SAM. We also summarize massive other amazing applications of SAM in vision and beyond. Finally, we maintain a continuously updated paper list and an open-source project summary for foundation model SAM at hrefhttps://github.com/liliu-avril/Awesome-Segment-Anythingcolormagentahere.