# Practical Region-level Attack against Segment Anything Models

arXiv: https://arxiv.org/abs/2404.08255 · HTML: https://mlsanity.com/paper/2404.08255/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 50 references that are in the map.
- First posted on arXiv 12 Apr 2024.

## Builds on (most cited first, 30 of 44)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Ensemble Adversarial Training: Attacks and Defenses](https://mlsanity.com/paper/1705.07204/) (2017)
- [Improving Transferability of Adversarial Examples with Input Diversity](https://mlsanity.com/paper/1803.06978/) (2018)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [Countering Adversarial Images using Input Transformations](https://mlsanity.com/paper/1711.00117/) (2017)
- [Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attacks](https://mlsanity.com/paper/1904.02884/) (2019)
- [Mitigating Adversarial Effects Through Randomization](https://mlsanity.com/paper/1711.01991/) (2017)
- [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023)
- [Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attacks](https://mlsanity.com/paper/1908.06281/) (2019)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [Defense against Adversarial Attacks Using High-Level Representation Guided Denoiser](https://mlsanity.com/paper/1712.02976/) (2017)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [Fast Segment Anything](https://mlsanity.com/paper/2306.12156/) (2023)
- [Adversarial Examples for Semantic Segmentation and Object Detection](https://mlsanity.com/paper/1703.08603/) (2017)
- [Track Anything: Segment Anything Meets Videos](https://mlsanity.com/paper/2304.11968/) (2023)
- [VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking](https://mlsanity.com/paper/2303.11301/) (2023)
- [EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything](https://mlsanity.com/paper/2312.00863/) (2023)
- [Semantic-SAM: Segment and Recognize Anything at Any Granularity](https://mlsanity.com/paper/2307.04767/) (2023)
- [Robustness via curvature regularization, and vice versa](https://mlsanity.com/paper/1811.09716/) (2018)
- [Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNets](https://mlsanity.com/paper/2002.05990/) (2020)
- [Frequency Domain Model Augmentation for Adversarial Attack](https://mlsanity.com/paper/2207.05382/) (2022)

## Built on by (most cited first, 9 of 9)

- [Vanish into Thin Air: Cross-prompt Universal Adversarial Attacks for SAM2](https://mlsanity.com/paper/2510.24195/) (2025)
- [Region-Guided Attack on the Segment Anything Model (SAM)](https://mlsanity.com/paper/2411.02974/) (2024)
- [FusionSAM: Visual Multi-Modal Learning with Segment Anything](https://mlsanity.com/paper/2408.13980/) (2024)
- [Creating Blank Canvas Against AI-enabled Image Forgery](https://mlsanity.com/paper/2511.22237/) (2025)
- [Proxy-Embedding as an Adversarial Teacher: An Embedding-Guided Bidirectional Attack for Referring Expression Segmentation Models](https://mlsanity.com/paper/2506.16157/) (2025)
- [SAM Encoder Breach by Adversarial Simplicial Complex Triggers Downstream Model Failures](https://mlsanity.com/paper/2508.06127/) (2025)
- [ChronoVision: Temporal Reasoning via Latent State Reconstruction](https://mlsanity.com/paper/2608.05631/) (2026)
- [Universal Concept Disruption for SAM3 Image Segmentation](https://mlsanity.com/paper/2608.05983/) (2026)
- [Universal Cross-Prompt Adversarial Attacks on Promptable Concept Segmentation](https://mlsanity.com/paper/2609.39265/) (2026)

## Abstract (arXiv)

Segment Anything Models (SAM) have made significant advancements in image segmentation, allowing users to segment target portions of an image with a single click (i.e., user prompt). Given its broad applications, the robustness of SAM against adversarial attacks is a critical concern. While recent works have explored adversarial attacks against a pre-defined prompt/click, their threat model is not yet realistic: (1) they often assume the user-click position is known to the attacker (point-based attack), and (2) they often operate under a white-box setting with limited transferability. In this paper, we propose a more practical region-level attack where attackers do not need to know the precise user prompt. The attack remains effective as the user clicks on any point on the target object in the image, hiding the object from SAM. Also, by adapting a spectrum transformation method, we make the attack more transferable under a black-box setting. Both control experiments and testing against real-world SAM services confirm its effectiveness.