mlsanityData to 7 Oct 2026

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP

arXiv 2505.05528 ↗First posted 8 May 2025cs.CV cs.AI cs.CLcited by 21

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

As Contrastive Language-Image Pre-training (CLIP) models are increasingly adopted for diverse downstream tasks and integrated into large vision-language models (VLMs), their susceptibility to adversarial perturbations has emerged as a critical concern. In this work, we introduce X-Transfer, a novel attack method that exposes a universal adversarial vulnerability in CLIP. X-Transfer generates a Universal Adversarial Perturbation (UAP) capable of deceiving various CLIP encoders and downstream VLMs across different samples, tasks, and domains. We refer to this property as super transferability--a single perturbation achieving cross-data, cross-domain, cross-model, and cross-task adversarial transferability simultaneously. This is achieved through surrogate scaling, a key innovation of our approach. Unlike existing methods that rely on fixed surrogate models, which are computationally intensive to scale, X-Transfer employs an efficient surrogate scaling strategy that dynamically selects a small subset of suitable surrogates from a large search space. Extensive evaluations demonstrate that X-Transfer significantly outperforms previous state-of-the-art UAP methods, establishing a new benchmark for adversarial transferability across CLIP models. The code is publicly available in our hrefhttps://github.com/HanxunH/XTransferBenchGitHub repository.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 122 of its 148 references that are in the map. First posted on arXiv 8 May 2025.

Builds on

122 of 148 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models11 Dec
  2. 2024Efficient and Effective Universal Adversarial Attack against Vision-Language Pre-training Models15 Oct
  3. 2024One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models8 Jun
  4. 2024White-box Multimodal Jailbreaks Against Large Vision-Language Models28 May
  5. 2024Universal Adversarial Perturbations for Vision-Language Pre-trained Models9 May
  6. 2024Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectory19 Mar
  7. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Image5 Mar
  8. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  9. 2024Test-Time Backdoor Attacks on Multimodal Large Language Models13 Feb
  10. 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation8 Dec
  11. 2023BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP26 Nov
  12. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
5 more
  1. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul
  2. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  3. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
  4. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec
  5. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 105

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  10. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  11. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2013Intriguing properties of neural networkscited by 5,114
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2022A ConvNet for the 2020scited by 3,545
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  11. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  13. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  14. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  15. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  16. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  17. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  18. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  19. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  20. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  21. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  22. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  23. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  24. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  25. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  26. 2018Robustness May Be at Odds with Accuracycited by 873
  27. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  28. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  29. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  30. 2016Universal adversarial perturbationscited by 774
  31. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  32. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  33. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 682
  34. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  35. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  36. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 591
  37. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  38. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 530

50 most cited of 105 All 105 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 13.

21 papers.

Sort:

By area:

  1. 2025DualTAP: A Dual-Task Adversarial Protector for Mobile MLLM Agentscited by 7
  2. 2025BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Modelscited by 5
  3. 2025TRAP: Targeted Redirecting of Agentic Preferencescited by 4
  4. 2026Toward Universal and Transferable Jailbreak Attacks on Vision-Language Modelscited by 4
  5. 2025CAVALRY-V: A Large-Scale Generator Framework for Adversarial Attacks on Video MLLMscited by 3
  6. 2025Poisoning Prompt-Guided Sampling in Video Large Language Modelscited by 3
  7. 2025HV-Attack: Hierarchical Visual Attack for Multimodal Retrieval Augmented Generationcited by 3
  8. 2025VISOR++: Universal Visual Inputs based Steering for Large Vision Language Modelscited by 1
  9. 2026Universal Adversarial Attacks against Closed-Source MLLMs via Target-View Routed Meta Optimizationcited by 1
  10. 2026Right Predictions, Misleading Explanations: On the Vulnerability of Vision-Language Model Explanationscited by 1
  11. 2026XPlainVerse: A Million-Scale Benchmark for Explainable Deepfake Detectioncited by 1
  12. 2025Cross-Modal Content Optimization for Steering Web Agent Preferencescited by 0
9 more
  1. 2026Stage-wise Attention-Guided Region Sequencing for Adversarial Attacks on Large Vision-Language Modelscited by 0
  2. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Modelscited by 0
  3. 2026Revealing Physical-World Semantic Vulnerabilities: Universal Adversarial Patch for Infrared Vision-Language Modelscited by 0
  4. 2026DarkLLM: Learning Language-Driven Adversarial Attacks with Large Language Modelscited by 0
  5. 2026Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguardscited by 0
  6. 2026Generating Multi-view Adversarial Examples for Visual Geometry Grounded Transformercited by 0
  7. 2026One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMscited by 0
  8. 2026VEX-Bench: Benchmarking Verification Complexity of LLM-Generated Misinformationcited by 0
  9. 2026Does Target Alignment Mean Target Recovery? An Evidence-Ladder Study of Adversarial Claims on Contrastive Encoderscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Improving Adversarial Transferability on Vision-Language Pre-training Models via Surrogate-Specific Bias Correction
  2. 2025Revisiting CroPA: A Reproducibility Study and Enhancements for Cross-Prompt Adversarial Transferability in Vision-Language Models
  3. 2025Bridging the Task Gap: Multi-Task Adversarial Transferability in CLIP and Its Derivatives
  4. 2026Proxy Robustness in Vision Language Models is Effortlessly Transferable
  5. 2024Enhancing Cross-Prompt Transferability in Vision-Language Models through Contextual Injection of Target Tokens
  6. 2024Prompt-Driven Contrastive Learning for Transferable Adversarial Attacks
  7. 2024An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Models
  8. 2025One Surrogate to Fool Them All: Universal, Transferable, and Targeted Adversarial Attacks with CLIP