mlsanityData to 14 Oct 2026

Adversarial Attacks on Foundational Vision Models

arXiv 2308.14597 ↗First posted 28 Aug 2023cs.CV cs.CR cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Rapid progress is being made in developing large, pretrained, task-agnostic foundational vision models such as CLIP, ALIGN, DINOv2, etc. In fact, we are approaching the point where these models do not have to be finetuned downstream, and can simply be used in zero-shot or with a lightweight probing head. Critically, given the complexity of working at this scale, there is a bottleneck where relatively few organizations in the world are executing the training then sharing the models on centralized platforms such as HuggingFace and torch.hub. The goal of this work is to identify several key adversarial vulnerabilities of these models in an effort to make future designs more robust. Intuitively, our attacks manipulate deep feature representations to fool an out-of-distribution (OOD) detector which will be required when using these open-world-aware models to solve closed-set downstream tasks. Our methods reliably make in-distribution (ID) images (w.r.t. a downstream task) be predicted as OOD and vice versa while existing in extremely low-knowledge-assumption threat models. We show our attacks to be potent in whitebox and blackbox settings, as well as when transferred across foundational model types (e.g., attack DINOv2 with CLIP)! This work is only just the beginning of a long journey towards adversarially robust foundational vision models.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 43 of its 53 references that are in the map. First posted on arXiv 28 Aug 2023.

Builds on

43 of 53 references.

References · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2023Segment Anythingcited by 8,234
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  8. 2013Intriguing properties of neural networkscited by 5,155
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2022A ConvNet for the 2020scited by 3,579
  12. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
31 more
  1. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  2. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  4. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  5. 2017Ensemble Adversarial Training: Attacks and Defensescited by 840
  6. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 599
  7. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 536
  8. 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465
  9. 2019Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attackscited by 442
  10. 2023Scaling Open-Vocabulary Object Detectioncited by 343
  11. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 290
  12. 2023A Cookbook of Self-Supervised Learningcited by 250
  13. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 220
  14. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 216
  15. 2023OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detectioncited by 179
  16. 2022Towards Adversarial Attack on Vision-Language Pre-training Modelscited by 144
  17. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Modelscited by 117
  18. 2020ATOM: Robustifying Out-of-distribution Detection Using Outlier Miningcited by 116
  19. 2022Revisiting Weakly Supervised Pre-Training of Visual Perception Modelscited by 116
  20. 2023Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modelingcited by 94
  21. 2020Robust Out-of-distribution Detection for Neural Networkscited by 57
  22. 2020Perturbing Across the Feature Hierarchy to Improve Standard and Strict Blackbox Attack Transferabilitycited by 57
  23. 2020Transferable Perturbations of Deep Feature Distributionscited by 51
  24. 2021Reading Isn't Believing: Adversarial Attacks On Multi-Modal Neuronscited by 29
  25. 2023Language-Driven Anchors for Zero-Shot Adversarial Robustnesscited by 23
  26. 2022Adversarial vulnerability of powerful near out-of-distribution detectioncited by 16
  27. 2022Pre-trained Adversarial Perturbationscited by 16
  28. 2022Fine-grain Inference on Out-of-Distribution Data with Hierarchical Classificationcited by 10
  29. 2020Generating Out of Distribution Adversarial Attack using Latent Space Poisoningcited by 5
  30. 2022Towards Out-of-Distribution Adversarial Robustnesscited by 5
  31. 2021Can Targeted Adversarial Examples Transfer When the Source and Target Models Have No Label Space Overlap?cited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 5; 2026: 3.

13 papers.

Sort:

By area:

  1. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustnesscited by 48
  2. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Modelscited by 48
  3. 2025Improving Generalizability and Undetectability for Targeted Adversarial Attacks on Multimodal Pre-trained Modelscited by 5
  4. 2024TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Abilitycited by 2
  5. 2025Robustness Tokens: Towards Adversarial Robustness of Transformerscited by 2
  6. 2025Universal and Transferable Attacks on Pathology Foundation Modelscited by 2
  7. 2024Liveness Detection in Computer Vision: Transformer-based Self-Supervised Learning for Face Anti-Spoofingcited by 1
  8. 2024Replace-then-Perturb: Targeted Adversarial Attacks With Visual Reasoning for Vision-Language Modelscited by 1
  9. 2025Task-Agnostic Attacks Against Vision Foundation Modelscited by 1
  10. 2026Hierarchically Robust Zero-shot Vision-language Modelscited by 1
  11. 2025MoAPT: Mixture of Adversarial Prompt Tuning for Vision-Language Modelscited by 0
  12. 2026Robotics-Inspired Guardrails for Foundation Models in Socially Sensitive Domainscited by 0
1 more
  1. 2026Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Modelcited by 0