# An Empirical Study on the Robustness of the Segment Anything Model (SAM)

arXiv: https://arxiv.org/abs/2305.06422 · HTML: https://mlsanity.com/paper/2305.06422/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 10 May 2023.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Images](https://mlsanity.com/paper/1805.06561/) (2018)
- [K-Net: Towards Unified Image Segmentation](https://mlsanity.com/paper/2106.14855/) (2021)
- [Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging](https://mlsanity.com/paper/2304.04155/) (2023)
- [Segment Anything Is Not Always Perfect: An Investigation of SAM on Different Real-world Applications](https://mlsanity.com/paper/2304.05750/) (2023)
- [Benchmarking the Robustness of Semantic Segmentation Models](https://mlsanity.com/paper/1908.05005/) (2019)
- [DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation](https://mlsanity.com/paper/1707.00652/) (2017)
- [Can SAM Segment Anything? When SAM Meets Camouflaged Object Detection](https://mlsanity.com/paper/2304.04709/) (2023)
- [SAM Struggles in Concealed Scenes -- Empirical Study on Segment Anything](https://mlsanity.com/paper/2304.06022/) (2023)
- [Polyp-SAM: Transfer SAM for Polyp Segmentation](https://mlsanity.com/paper/2305.00293/) (2023)
- [Can SAM Segment Polyps?](https://mlsanity.com/paper/2304.07583/) (2023)
- [Are Large Language Models Ready for Healthcare? A Comparative Study on Clinical Language Understanding](https://mlsanity.com/paper/2304.05368/) (2023)
- [Instance-Level Segmentation for Autonomous Driving with Deep Densely Connected MRFs](https://mlsanity.com/paper/1512.06735/) (2015)

## Built on by (most cited first, 23 of 23)

- [A Comprehensive Survey on Segment Anything Model for Vision and Beyond](https://mlsanity.com/paper/2305.08196/) (2023)
- [Metacognitive Prompting Improves Understanding in Large Language Models](https://mlsanity.com/paper/2308.05342/) (2023)
- [A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineering](https://mlsanity.com/paper/2306.06211/) (2023)
- [RobustSAM: Segment Anything Robustly on Degraded Images](https://mlsanity.com/paper/2406.09627/) (2024)
- [Robustness of SAM: Segment Anything Under Corruptions and Beyond](https://mlsanity.com/paper/2306.07713/) (2023)
- [On the Robustness of Segment Anything](https://mlsanity.com/paper/2305.16220/) (2023)
- [Robust Box Prompt based SAM for Medical Image Segmentation](https://mlsanity.com/paper/2407.21284/) (2024)
- [Unsegment Anything by Simulating Deformation](https://mlsanity.com/paper/2404.02585/) (2024)
- [SAM Meets UAP: Attacking Segment Anything Model With Universal Adversarial Perturbation](https://mlsanity.com/paper/2310.12431/) (2023)
- [Red-Teaming Segment Anything Model](https://mlsanity.com/paper/2404.02067/) (2024)
- [SALT: A Flexible Semi-Automatic Labeling Tool for General LiDAR Point Clouds with Cross-Scene Adaptability and 4D Consistency](https://mlsanity.com/paper/2503.23980/) (2025)
- [Robust SAM: On the Adversarial Robustness of Vision Foundation Models](https://mlsanity.com/paper/2504.08906/) (2025)
- [Segment Any-Quality Images with Generative Latent Space Enhancement](https://mlsanity.com/paper/2503.12507/) (2025)
- [TETRIS: Towards Exploring the Robustness of Interactive Segmentation](https://mlsanity.com/paper/2402.06132/) (2024)
- [RoSe: Robust Self-supervised Stereo Matching under Adverse Weather Conditions](https://mlsanity.com/paper/2509.19165/) (2025)
- [The SAM2-to-SAM3 Gap in the Segment Anything Model Family: Why Prompt-Based Expertise Fails in Concept-Driven Image Segmentation](https://mlsanity.com/paper/2512.06032/) (2025)
- [BALD-SAM: Disagreement-based Active Prompting in Interactive Segmentation](https://mlsanity.com/paper/2603.10828/) (2026)
- [Contour Field based Elliptical Shape Prior for the Segment Anything Model](https://mlsanity.com/paper/2504.12556/) (2025)
- [synth-dacl: Does Synthetic Defect Data Enhance Segmentation Accuracy and Robustness for Real-World Bridge Inspections?](https://mlsanity.com/paper/2506.14255/) (2025)
- [Semantic-E2VID: a Semantic-Enriched Paradigm for Event-to-Video Reconstruction](https://mlsanity.com/paper/2510.17347/) (2025)
- [Towards Integrating Uncertainty for Domain-Agnostic Segmentation](https://mlsanity.com/paper/2512.23427/) (2025)
- [Towards Any-Quality Image Segmentation via Generative and Adaptive Latent Space Enhancement](https://mlsanity.com/paper/2601.02018/) (2026)
- [BREPS: Bounding-Box Robustness Evaluation of Promptable Segmentation](https://mlsanity.com/paper/2601.15123/) (2026)

## Abstract (arXiv)

The Segment Anything Model (SAM) is a foundation model for general image segmentation. Although it exhibits impressive performance predominantly on natural images, understanding its robustness against various image perturbations and domains is critical for real-world applications where such challenges frequently arise. In this study we conduct a comprehensive robustness investigation of SAM under diverse real-world conditions. Our experiments encompass a wide range of image perturbations. Our experimental results demonstrate that SAM's performance generally declines under perturbed images, with varying degrees of vulnerability across different perturbations. By customizing prompting techniques and leveraging domain knowledge based on the unique characteristics of each dataset, the model's resilience to these perturbations can be enhanced, addressing dataset-specific challenges. This work sheds light on the limitations and strengths of SAM in real-world applications, promoting the development of more robust and versatile image segmentation solutions.