mlsanityData to 7 Oct 2026

Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models

arXiv 2402.12336 ↗First posted 19 Feb 2024cs.LG cs.AI cs.CVcited by 119

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Multi-modal foundation models like OpenFlamingo, LLaVA, and GPT-4 are increasingly used for various real-world tasks. Prior work has shown that these models are highly vulnerable to adversarial attacks on the vision modality. These attacks can be leveraged to spread fake information or defraud users, and thus pose a significant risk, which makes the robustness of large multi-modal foundation models a pressing problem. The CLIP model, or one of its variants, is used as a frozen vision encoder in many large vision-language models (LVLMs), e.g. LLaVA and OpenFlamingo. We propose an unsupervised adversarial fine-tuning scheme to obtain a robust CLIP vision encoder, which yields robustness on all vision down-stream tasks (LVLMs, zero-shot classification) that rely on CLIP. In particular, we show that stealth-attacks on users of LVLMs by a malicious third party providing manipulated images are no longer possible once one replaces the original CLIP model with our robust one. No retraining or fine-tuning of the down-stream LVLMs is required. The code and robust models are available at https://github.com/chs20/RobustVLM

Key facts. Cited by 119 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 68 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

51 of 68 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast13 Feb
  2. 2023How Robust is Google's Bard to Adversarial Image Attacks?21 Sep
  3. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul
  4. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
  5. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2013Intriguing properties of neural networkscited by 5,114
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
34 more
  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2013Describing Textures in the Wildcited by 2,110
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  6. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  8. 2019Towards VQA Models That Can Readcited by 1,696
  9. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  10. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  11. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  12. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  13. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  14. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  15. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  16. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  17. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  18. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  19. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  20. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  21. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 165
  22. 2020Robust Pre-Training by Adversarial Contrastive Learningcited by 144
  23. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 142
  24. 2020Adversarial Self-Supervised Contrastive Learningcited by 138
  25. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129
  26. 2023Revisiting Adversarial Training for ImageNet: Architectures, Training and Generalization across Threat Modelscited by 79
  27. 2019Metric Learning for Adversarial Robustnesscited by 77
  28. 2021When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?cited by 76
  29. 2022Decoupled Adversarial Contrastive Learning for Self-supervised Adversarial Robustnesscited by 20
  30. 2023Rethinking the Effect of Data Augmentation in Adversarial Contrastive Learningcited by 20
  31. 2022Pre-trained Adversarial Perturbationscited by 16
  32. 2022Enhancing Adversarial Robustness for Deep Metric Learningcited by 10
  33. 2023Enhancing Adversarial Contrastive Learning via Adversarial Invariant Regularizationcited by 10
  34. 2021Adversarial Attack and Defense in Deep Rankingcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 55; 2026: 46.

119 papers.

Sort:

By area:

  1. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  2. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  3. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Drivingcited by 35
  4. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuningcited by 33
  5. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 28
  6. 2024B-AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Black-box Adversarial Visual-Instructionscited by 26
  7. 2024Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Modelscited by 23
  8. 2025CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIPcited by 23
  9. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Modelscited by 22
  10. 2025X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIPcited by 21
  11. 2024VLMGuard: Bootstrapping Malicious Prompt Detectors from Unlabeled Vision-Language Prompts in the Wildcited by 17
  12. 2025Survey of Adversarial Robustness in Multimodal Large Language Modelscited by 14
38 more
  1. 2025Enhancing CLIP Robustness via Cross-Modality Alignmentcited by 14
  2. 2024Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attackscited by 13
  3. 2025Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Modelscited by 13
  4. 2025DECEPTICON: How Dark Patterns Manipulate Web Agentscited by 13
  5. 2024Sim-CLIP: Unsupervised Siamese Adversarial Fine-Tuning for Robust and Semantically-Rich Vision-Language Modelscited by 12
  6. 2025E2AT: Multimodal Jailbreak Defense via Dynamic Joint Optimization for Multimodal Large Language Modelscited by 12
  7. 2025On the Limitations of Vision-Language Models in Understanding Image Transformscited by 12
  8. 2025MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agentscited by 12
  9. 2025Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Modelscited by 12
  10. 2024As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?cited by 11
  11. 2025MMT-ARD: Multimodal Multi-Teacher Adversarial Distillation for Robust Vision-Language Modelscited by 11
  12. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustnesscited by 10
  13. 2025Adversarially Robust CLIP Models Can Induce Better (Robust) Perceptual Metricscited by 10
  14. 2024Omniview-Tuning: Boosting Viewpoint Invariance of Vision-Language Pre-training Modelscited by 9
  15. 2024Adaptive Concept Bottleneck for Foundation Models Under Distribution Shiftscited by 9
  16. 2025Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Modelscited by 9
  17. 2025V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMscited by 9
  18. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  19. 2025AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimizationcited by 8
  20. 2025VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Modelscited by 7
  21. 2024Adversarial Robustness of AI-Generated Image Detectors in the Real Worldcited by 6
  22. 2024Adversarial Prompt Distillation for Vision-Language Modelscited by 6
  23. 2025OCRT: Boosting Foundation Models in the Open World with Object-Concept-Relation Triadcited by 6
  24. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniquescited by 5
  25. 2024Not all Views are Created Equal: Analyzing Viewpoint Instabilities in Vision Foundation Modelscited by 5
  26. 2025Text-to-Decision Agent: Offline Meta-Reinforcement Learning from Natural Language Supervisioncited by 5
  27. 2025Enhancing Targeted Adversarial Attacks on Large Vision-Language Models via Intermediate Projectorcited by 5
  28. 2025Improving Generalizability and Undetectability for Targeted Adversarial Attacks on Multimodal Pre-trained Modelscited by 5
  29. 2026Reclaiming Lost Text Layers for Source-Free Cross-Domain Few-Shot Learningcited by 5
  30. 2025LeakyCLIP: Extracting Training Data from CLIPcited by 4
  31. 2025Unveiling Trust in Multimodal Large Language Models: Evaluation, Analysis, and Mitigationcited by 4
  32. 2026Is the Modality Gap a Bug or a Feature? A Robustness Perspectivecited by 4
  33. 2025CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Modelscited by 3
  34. 2025Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-based Attackscited by 3
  35. 2025Bootstrapping LLM Robustness for VLM Safety via Reducing the Pretraining Modality Gapcited by 3
  36. 2025Processing and acquisition traces in visual encoders: What does CLIP know about your camera?cited by 3
  37. 2025ToxicTextCLIP: Text-Based Poisoning and Backdoor Attacks on CLIP Pre-trainingcited by 3
  38. 2026PA-Attack: Guiding Gray-Box Attacks on LVLM Vision Encoders with Prototypes and Attentioncited by 3

50 most cited of 119 All 119 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Investigating Adversarial Robustness of Multi-modal Large Language Models
  2. 2025Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models
  3. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective
  4. 2026Proxy Robustness in Vision Language Models is Effortlessly Transferable
  5. 2025Few-Shot Adversarial Low-Rank Fine-Tuning of Vision-Language Models
  6. 2024A Hybrid Defense Strategy for Boosting Adversarial Robustness in Vision-Language Models
  7. 2025Quality Text, Robust Vision: The Role of Language in Enhancing Visual Robustness of Vision-Language Models
  8. 2024Toward a Holistic Evaluation of Robustness in CLIP Models