# DarkSAM: Fooling Segment Anything Model to Segment Nothing

arXiv: https://arxiv.org/abs/2409.17874 · HTML: https://mlsanity.com/paper/2409.17874/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 45 references that are in the map.
- First posted on arXiv 26 Sep 2024.

## Builds on (most cited first, 30 of 38)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Universal adversarial perturbations](https://mlsanity.com/paper/1610.08401/) (2016)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [Personalize Segment Anything Model with One Shot](https://mlsanity.com/paper/2305.03048/) (2023)
- [Track Anything: Segment Anything Meets Videos](https://mlsanity.com/paper/2304.11968/) (2023)
- [Semantic-SAM: Segment and Recognize Anything at Any Granularity](https://mlsanity.com/paper/2307.04767/) (2023)
- [A Self-supervised Approach for Adversarial Robustness](https://mlsanity.com/paper/2006.04924/) (2020)
- [On the Robustness of Semantic Segmentation Models to Adversarial Attacks](https://mlsanity.com/paper/1711.09856/) (2017)
- [AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learning](https://mlsanity.com/paper/2308.07026/) (2023)
- [Universal Adversarial Perturbations Against Semantic Image Segmentation](https://mlsanity.com/paper/1704.05712/) (2017)
- [Generalizable Data-free Objective for Crafting Universal Adversarial Perturbations](https://mlsanity.com/paper/1801.08092/) (2018)
- [MA-SAM: Modality-agnostic SAM Adaptation for 3D Medical Image Segmentation](https://mlsanity.com/paper/2309.08842/) (2023)
- [SegPGD: An Effective and Efficient Adversarial Attack for Evaluating and Boosting Segmentation Robustness](https://mlsanity.com/paper/2207.12391/) (2022)
- [Fast Feature Fool: A data independent approach to universal adversarial perturbations](https://mlsanity.com/paper/1707.05572/) (2017)
- [Polyp-SAM: Transfer SAM for Polyp Segmentation](https://mlsanity.com/paper/2305.00293/) (2023)
- [NAG: Network for Adversary Generation](https://mlsanity.com/paper/1712.03390/) (2017)
- [SAM-guided Graph Cut for 3D Instance Segmentation](https://mlsanity.com/paper/2312.08372/) (2023)

## Built on by (most cited first, 30 of 32)

- [ADVEDM:Fine-grained Adversarial Attack against VLM-based Embodied Agents](https://mlsanity.com/paper/2509.16645/) (2025)
- [Transferable Adversarial Attacks on SAM and Its Downstream Models](https://mlsanity.com/paper/2410.20197/) (2024)
- [X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP](https://mlsanity.com/paper/2505.05528/) (2025)
- [Poison-splat: Computation Cost Attack on 3D Gaussian Splatting](https://mlsanity.com/paper/2410.08190/) (2024)
- [NumbOD: A Spatial-Frequency Fusion Attack Against Object Detectors](https://mlsanity.com/paper/2412.16955/) (2024)
- [Unlearnable 3D Point Clouds: Class-wise Transformation Is All You Need](https://mlsanity.com/paper/2410.03644/) (2024)
- [PB-UAP: Hybrid Universal Adversarial Attack For Image Segmentation](https://mlsanity.com/paper/2412.16651/) (2024)
- [When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision-Language-Action Models](https://mlsanity.com/paper/2511.21192/) (2025)
- [Vanish into Thin Air: Cross-prompt Universal Adversarial Attacks for SAM2](https://mlsanity.com/paper/2510.24195/) (2025)
- [Towards Reliable Forgetting: A Survey on Machine Unlearning Verification](https://mlsanity.com/paper/2506.15115/) (2025)
- [SegTrans: Transferable Adversarial Examples for Segmentation Models](https://mlsanity.com/paper/2510.08922/) (2025)
- [ForensicsSAM: Toward Robust and Unified Image Forgery Detection and Localization Resisting to Adversarial Attack](https://mlsanity.com/paper/2508.07402/) (2025)
- [DarkHash: A Data-Free Backdoor Attack Against Deep Hashing](https://mlsanity.com/paper/2510.08094/) (2025)
- [Time Is All It Takes: Spike-Retiming Attacks on Event-Driven Spiking Neural Networks](https://mlsanity.com/paper/2602.03284/) (2026)
- [Improving Generalization of Universal Adversarial Perturbation via Dynamic Maximin Optimization](https://mlsanity.com/paper/2503.12793/) (2025)
- [Mix-QSAM: Mixed-Precision Quantization of the Segment Anything Model](https://mlsanity.com/paper/2505.04861/) (2025)
- [Erosion Attack for Adversarial Training to Enhance Semantic Segmentation Robustness](https://mlsanity.com/paper/2601.14950/) (2026)
- [Region-Guided Attack on the Segment Anything Model (SAM)](https://mlsanity.com/paper/2411.02974/) (2024)
- [UFVideo: Towards Unified Fine-Grained Video Cooperative Understanding with Large Language Models](https://mlsanity.com/paper/2512.11336/) (2025)
- [Towards Real-World Deepfake Detection: A Diverse In-the-wild Dataset of Forgery Faces](https://mlsanity.com/paper/2510.08067/) (2025)
- [Dual-View Inference Attack: Machine Unlearning Amplifies Privacy Exposure](https://mlsanity.com/paper/2512.16126/) (2025)
- [Transferable Physical-World Adversarial Patches Against Object Detection in Autonomous Driving](https://mlsanity.com/paper/2604.23105/) (2026)
- [LSP-ST: Ladder Shape-Biased Side-Tuning for Robust Infrared Small Target Detection](https://mlsanity.com/paper/2504.14481/) (2025)
- [SAM Encoder Breach by Adversarial Simplicial Complex Triggers Downstream Model Failures](https://mlsanity.com/paper/2508.06127/) (2025)
- [Malicious Image Analysis via Vision-Language Segmentation Fusion: Detection, Element, and Location in One-shot](https://mlsanity.com/paper/2512.04599/) (2025)
- [Low-Cost Hard-Label Adversarial Attack with Theoretical Foundations](https://mlsanity.com/paper/2601.14300/) (2026)
- [BREPS: Bounding-Box Robustness Evaluation of Promptable Segmentation](https://mlsanity.com/paper/2601.15123/) (2026)
- [DarkLLM: Learning Language-Driven Adversarial Attacks with Large Language Models](https://mlsanity.com/paper/2605.18868/) (2026)
- [VFACamou: View-Fused Adversarial Camouflage for Environment-Adaptive Physical Evasion](https://mlsanity.com/paper/2606.19736/) (2026)
- [Universal Concept Disruption for SAM3 Image Segmentation](https://mlsanity.com/paper/2608.05983/) (2026)

## Abstract (arXiv)

Segment Anything Model (SAM) has recently gained much attention for its outstanding generalization to unseen data and tasks. Despite its promising prospect, the vulnerabilities of SAM, especially to universal adversarial perturbation (UAP) have not been thoroughly investigated yet. In this paper, we propose DarkSAM, the first prompt-free universal attack framework against SAM, including a semantic decoupling-based spatial attack and a texture distortion-based frequency attack. We first divide the output of SAM into foreground and background. Then, we design a shadow target strategy to obtain the semantic blueprint of the image as the attack target. DarkSAM is dedicated to fooling SAM by extracting and destroying crucial object features from images in both spatial and frequency domains. In the spatial domain, we disrupt the semantics of both the foreground and background in the image to confuse SAM. In the frequency domain, we further enhance the attack effectiveness by distorting the high-frequency components (i.e., texture information) of the image. Consequently, with a single UAP, DarkSAM renders SAM incapable of segmenting objects across diverse images with varying prompts. Experimental results on four datasets for SAM and its two variant models demonstrate the powerful attack capability and transferability of DarkSAM.