# Universal Cross-Prompt Adversarial Attacks on Promptable Concept Segmentation

arXiv: https://arxiv.org/abs/2609.39265 · HTML: https://mlsanity.com/paper/2609.39265/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 37 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018)
- [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023)
- [MOSE: A New Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2302.01872/) (2023)
- [AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learning](https://mlsanity.com/paper/2308.07026/) (2023)
- [Deflecting Adversarial Attacks with Pixel Deflection](https://mlsanity.com/paper/1801.08926/) (2018)
- [Downstream-agnostic Adversarial Examples](https://mlsanity.com/paper/2307.12280/) (2023)
- [DarkSAM: Fooling Segment Anything Model to Segment Nothing](https://mlsanity.com/paper/2409.17874/) (2024)
- [Robustness of SAM: Segment Anything Under Corruptions and Beyond](https://mlsanity.com/paper/2306.07713/) (2023)
- [X-SAM: From Segment Anything to Any Segmentation](https://mlsanity.com/paper/2508.04655/) (2025)
- [Securely Fine-tuning Pre-trained Encoders Against Adversarial Examples](https://mlsanity.com/paper/2403.10801/) (2024)
- [Transferable Adversarial Attacks on SAM and Its Downstream Models](https://mlsanity.com/paper/2410.20197/) (2024)
- [NumbOD: A Spatial-Frequency Fusion Attack Against Object Detectors](https://mlsanity.com/paper/2412.16955/) (2024)
- [Is SAM 2 Better than SAM in Medical Image Segmentation?](https://mlsanity.com/paper/2408.04212/) (2024)
- [Black-box Targeted Adversarial Attack on Segment Anything (SAM)](https://mlsanity.com/paper/2310.10010/) (2023)
- [Vanish into Thin Air: Cross-prompt Universal Adversarial Attacks for SAM2](https://mlsanity.com/paper/2510.24195/) (2025)
- [Practical Region-level Attack against Segment Anything Models](https://mlsanity.com/paper/2404.08255/) (2024)
- [SegTrans: Transferable Adversarial Examples for Segmentation Models](https://mlsanity.com/paper/2510.08922/) (2025)
- [Unsegment Anything by Simulating Deformation](https://mlsanity.com/paper/2404.02585/) (2024)
- [Segment (Almost) Nothing: Prompt-Agnostic Adversarial Attacks on Segmentation Models](https://mlsanity.com/paper/2311.14450/) (2023)
- [SAM Meets UAP: Attacking Segment Anything Model With Universal Adversarial Perturbation](https://mlsanity.com/paper/2310.12431/) (2023)
- [Robust SAM: On the Adversarial Robustness of Vision Foundation Models](https://mlsanity.com/paper/2504.08906/) (2025)
- [EfficientSAM3: Progressive Hierarchical Distillation for Video Concept Segmentation from SAM1, 2, and 3](https://mlsanity.com/paper/2511.15833/) (2025)
- [Efficient-SAM2: Accelerating SAM2 with Object-Aware Visual Encoding and Memory Retrieval](https://mlsanity.com/paper/2602.08224/) (2026)

## Abstract (arXiv)

The Segment Anything Model (SAM) achieves remarkable performance in visual segmentation. The latest SAM3 extends promptable segmentation to concept-level prediction, broadening the scope of segmentation foundation models. While recent works reveal that SAM and SAM2 are vulnerable to adversarial examples, the robustness of SAM3 under the concept segmentation paradigm remains unexplored. In addition, existing adversarial attacks on SAM-series models exhibit limited cross-prompt transferability. To this end, we propose AdvPCS, a universal cross-prompt adversarial attack for Promptable Concept Segmentation (PCS), including a min-max prompt optimization strategy, a global-local perception deception attack, and a temporal transition deviation attack. Specifically, we first identify the hardest-to-attack prompts via min-max bilevel optimization. In the inner maximization, we enhance diversity over candidate point, box, and text prompts. In the outer minimization, we select prompts with the highest responses based on the confidence scores output by the detector. Given the selected prompts, we apply the perception deception attack to minimize both global and local existence probabilities under joint prompting and employ the temporal memory misalignment attack to maximize inter-frame semantic inconsistency and corrupt memory pointers. Extensive experiments on four benchmark datasets show that a single universal adversarial perturbation (UAP) generated by our method generalizes across frames from different videos and achieves strong attack performance under point, box, and text prompts. In particular, it reduces the average mIoU of various PCS models on the SA-CO dataset to below 5% under text prompts, demonstrating strong attack ability.