Task-Agnostic Attacks Against Vision Foundation Models
Not on the map. Why?
Abstract (arXiv)
The study of security in machine learning mainly focuses on downstream task-specific attacks, where the adversarial example is obtained by optimizing a loss function specific to the downstream task. At the same time, it has become standard practice for machine learning practitioners to adopt publicly available pre-trained vision foundation models, effectively sharing a common backbone architecture across a multitude of applications such as classification, segmentation, depth estimation, retrieval, question-answering and more. The study of attacks on such foundation models and their impact to multiple downstream tasks remains vastly unexplored. This work proposes a general framework that forges task-agnostic adversarial examples by maximally disrupting the feature representation obtained with foundation models. We extensively evaluate the security of the feature representations obtained by popular vision foundation models by measuring the impact of this attack on multiple downstream tasks and its transferability between models.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 58 of its 69 references that are in the map. First posted on arXiv 5 Mar 2025.
Builds on
58 of 69 references.
References · 58
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023Segment Anythingcited by 8,234
- 2023Visual Instruction Tuningcited by 7,869
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2014Explaining and Harnessing Adversarial Examplescited by 7,364
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2013Intriguing properties of neural networkscited by 5,155
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
38 more
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
- 2015VQA: Visual Question Answeringcited by 2,506
- 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
- 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
- 2024Depth Anything V2cited by 1,422
- 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,360
- 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,318
- 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
- 2021Grounded Language-Image Pre-trainingcited by 1,156
- 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
- 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
- 2024PaliGemma: A versatile 3B VLM for transfercited by 696
- 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
- 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
- 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 410
- 2022Scaling Language-Image Pre-training via Maskingcited by 317
- 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
- 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
- 2017Adversarial Examples for Semantic Segmentation and Object Detectioncited by 275
- 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
- 2018Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarkingcited by 191
- 2020Adversarial Self-Supervised Contrastive Learningcited by 139
- 2022(Certified!!) Adversarial Robustness for Free!cited by 132
- 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Modelscited by 117
- 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
- 2020RayS: A Ray Searching Method for Hard-label Adversarial Attackcited by 96
- 2021Watermarking Images in Self-Supervised Latent Spacescited by 94
- 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
- 2023Structure Invariant Transformation for better Adversarial Transferabilitycited by 62
- 2020SurFree: a fast surrogate-free black-box attackcited by 60
- 2021Fast Minimum-norm Adversarial Attacks through Adaptive Norm Constraintscited by 50
- 2024LocCa: Visual Pretraining with Location-aware Captionerscited by 35
- 2024Physical 3D Adversarial Attacks against Monocular Depth Estimation in Autonomous Drivingcited by 32
- 2020Adversarial Patch Attacks on Monocular Depth Estimation Networkscited by 29
- 2019Exact Adversarial Attack to Image Captioning via Structured Output Learning with Latent Variablescited by 26
- 2019Targeted Mismatch Adversarial Attack: Query with a Flower to Retrieve the Towercited by 26
50 most cited of 58 All 58 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.