# Red-Teaming Segment Anything Model

arXiv: https://arxiv.org/abs/2404.02067 · HTML: https://mlsanity.com/paper/2404.02067/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 43 of its 51 references that are in the map.
- First posted on arXiv 2 Apr 2024.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 30 of 43)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Universal adversarial perturbations](https://mlsanity.com/paper/1610.08401/) (2016)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [Simple Black-box Adversarial Attacks](https://mlsanity.com/paper/1905.07121/) (2019)
- [Glaze: Protecting Artists from Style Mimicry by Text-to-Image Models](https://mlsanity.com/paper/2302.04222/) (2023)
- [Neuron Shapley: Discovering the Responsible Neurons](https://mlsanity.com/paper/2002.09815/) (2020)
- [Machine Unlearning: Solutions and Challenges](https://mlsanity.com/paper/2308.07061/) (2023)
- [Maximal Jacobian-based Saliency Map Attack](https://mlsanity.com/paper/1808.07945/) (2018)
- [Adaptive Testing of Computer Vision Models](https://mlsanity.com/paper/2212.02774/) (2022)
- [Robustness of SAM: Segment Anything Under Corruptions and Beyond](https://mlsanity.com/paper/2306.07713/) (2023)
- [Towards Better Understanding Attribution Methods](https://mlsanity.com/paper/2205.10435/) (2022)
- [On the Robustness of Segment Anything](https://mlsanity.com/paper/2305.16220/) (2023)

## Built on by (most cited first, 4 of 4)

- [Aggregated Attributions for Explanatory Analysis of 3D Segmentation Models](https://mlsanity.com/paper/2407.16653/) (2024)
- [Robust SAM: On the Adversarial Robustness of Vision Foundation Models](https://mlsanity.com/paper/2504.08906/) (2025)
- [Geometric Red-Teaming for Robotic Manipulation](https://mlsanity.com/paper/2509.12379/) (2025)
- [Birds look like cars: Adversarial analysis of intrinsically interpretable deep learning](https://mlsanity.com/paper/2503.08636/) (2025)

## Abstract (arXiv)

Foundation models have emerged as pivotal tools, tackling many complex tasks through pre-training on vast datasets and subsequent fine-tuning for specific applications. The Segment Anything Model is one of the first and most well-known foundation models for computer vision segmentation tasks. This work presents a multi-faceted red-teaming analysis that tests the Segment Anything Model against challenging tasks: (1) We analyze the impact of style transfer on segmentation masks, demonstrating that applying adverse weather conditions and raindrops to dashboard images of city roads significantly distorts generated masks. (2) We focus on assessing whether the model can be used for attacks on privacy, such as recognizing celebrities' faces, and show that the model possesses some undesired knowledge in this task. (3) Finally, we check how robust the model is to adversarial attacks on segmentation masks under text prompts. We not only show the effectiveness of popular white-box attacks and resistance to black-box attacks but also introduce a novel approach - Focused Iterative Gradient Attack (FIGA) that combines white-box approaches to construct an efficient attack resulting in a smaller number of modified pixels. All of our testing methods and analyses indicate a need for enhanced safety measures in foundation models for image segmentation.