# Black-box Targeted Adversarial Attack on Segment Anything (SAM) arXiv: https://arxiv.org/abs/2310.10010 ยท HTML: https://mlsanity.com/paper/2310.10010/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Builds on 52 of its 77 references that are in the map. - First posted on arXiv 16 Oct 2023. ## Builds on (most cited first, 30 of 52) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021) - [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014) - [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017) - [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013) - [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021) - [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021) - [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019) - [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019) - [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016) - [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021) - [Evasion Attacks against Machine Learning at Test Time](https://mlsanity.com/paper/1708.06131/) (2017) - [Improving Transferability of Adversarial Examples with Input Diversity](https://mlsanity.com/paper/1803.06978/) (2018) - [Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks](https://mlsanity.com/paper/1704.01155/) (2017) - [Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attacks](https://mlsanity.com/paper/1904.02884/) (2019) - [A large annotated medical image dataset for the development and evaluation of segmentation algorithms](https://mlsanity.com/paper/1902.09063/) (2019) - [Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attacks](https://mlsanity.com/paper/1908.06281/) (2019) - [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023) - [Fast Segment Anything](https://mlsanity.com/paper/2306.12156/) (2023) - [Text-to-image Diffusion Models in Generative AI: A Survey](https://mlsanity.com/paper/2303.07909/) (2023) - [Track Anything: Segment Anything Meets Videos](https://mlsanity.com/paper/2304.11968/) (2023) - [A study of the effect of JPG compression on adversarial images](https://mlsanity.com/paper/1608.00853/) (2016) - [Data Augmentation Can Improve Robustness](https://mlsanity.com/paper/2111.05328/) (2021) - [MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining](https://mlsanity.com/paper/2208.12262/) (2022) - [Semantic-SAM: Segment and Recognize Anything at Any Granularity](https://mlsanity.com/paper/2307.04767/) (2023) - [Adversarial Robustness through Local Linearization](https://mlsanity.com/paper/1907.02610/) (2019) - [Can SAM Segment Anything? When SAM Meets Camouflaged Object Detection](https://mlsanity.com/paper/2304.04709/) (2023) ## Built on by (most cited first, 13 of 13) - [Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety](https://mlsanity.com/paper/2502.05206/) (2025) - [MobileSAMv2: Faster Segment Anything to Everything](https://mlsanity.com/paper/2312.09579/) (2023) - [Practical Region-level Attack against Segment Anything Models](https://mlsanity.com/paper/2404.08255/) (2024) - [Unsegment Anything by Simulating Deformation](https://mlsanity.com/paper/2404.02585/) (2024) - [ForensicsSAM: Toward Robust and Unified Image Forgery Detection and Localization Resisting to Adversarial Attack](https://mlsanity.com/paper/2508.07402/) (2025) - [UniUltra: Interactive Parameter-Efficient SAM2 for Universal Ultrasound Segmentation](https://mlsanity.com/paper/2511.15771/) (2025) - [Red-Teaming Segment Anything Model](https://mlsanity.com/paper/2404.02067/) (2024) - [Robust SAM: On the Adversarial Robustness of Vision Foundation Models](https://mlsanity.com/paper/2504.08906/) (2025) - [Region-Guided Attack on the Segment Anything Model (SAM)](https://mlsanity.com/paper/2411.02974/) (2024) - [SAM Encoder Breach by Adversarial Simplicial Complex Triggers Downstream Model Failures](https://mlsanity.com/paper/2508.06127/) (2025) - [DarkLLM: Learning Language-Driven Adversarial Attacks with Large Language Models](https://mlsanity.com/paper/2605.18868/) (2026) - [Universal Concept Disruption for SAM3 Image Segmentation](https://mlsanity.com/paper/2608.05983/) (2026) - [Universal Cross-Prompt Adversarial Attacks on Promptable Concept Segmentation](https://mlsanity.com/paper/2609.39265/) (2026) ## Abstract (arXiv) Deep recognition models are widely vulnerable to adversarial examples, which change the model output by adding quasi-imperceptible perturbation to the image input. Recently, Segment Anything Model (SAM) has emerged to become a popular foundation model in computer vision due to its impressive generalization to unseen data and tasks. Realizing flexible attacks on SAM is beneficial for understanding the robustness of SAM in the adversarial context. To this end, this work aims to achieve a targeted adversarial attack (TAA) on SAM. Specifically, under a certain prompt, the goal is to make the predicted mask of an adversarial example resemble that of a given target image. The task of TAA on SAM has been realized in a recent arXiv work in the white-box setup by assuming access to prompt and model, which is thus less practical. To address the issue of prompt dependence, we propose a simple yet effective approach by only attacking the image encoder. Moreover, we propose a novel regularization loss to enhance the cross-model transferability by increasing the feature dominance of adversarial images over random natural images. Extensive experiments verify the effectiveness of our proposed simple techniques to conduct a successful black-box TAA on SAM.