# Failures to Find Transferable Image Jailbreaks Between Vision-Language Models

arXiv: https://arxiv.org/abs/2407.15211 · HTML: https://mlsanity.com/paper/2407.15211/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 28 of its 115 references that are in the map.
- First posted on arXiv 21 Jul 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)
- LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/)

## Builds on (most cited first, 28 of 28)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Delving into Transferable Adversarial Examples and Black-box Attacks](https://mlsanity.com/paper/1611.02770/) (2016)
- [Safety Alignment Should Be Made More Than Just a Few Tokens Deep](https://mlsanity.com/paper/2406.05946/) (2024)
- [Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples](https://mlsanity.com/paper/1605.07277/) (2016)
- [Improving Alignment and Robustness with Circuit Breakers](https://mlsanity.com/paper/2406.04313/) (2024)
- [Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks](https://mlsanity.com/paper/2302.05733/) (2023)
- [A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity](https://mlsanity.com/paper/2401.01967/) (2024)
- [How Robust is Google's Bard to Adversarial Image Attacks?](https://mlsanity.com/paper/2309.11751/) (2023)
- [Model evaluation for extreme risks](https://mlsanity.com/paper/2305.15324/) (2023)
- [Image Hijacks: Adversarial Images can Control Generative Models at Runtime](https://mlsanity.com/paper/2309.00236/) (2023)
- [Defending Against Unforeseen Failure Modes with Latent Adversarial Training](https://mlsanity.com/paper/2403.05030/) (2024)
- [ImgTrojan: Jailbreaking Vision-Language Models with ONE Image](https://mlsanity.com/paper/2403.02910/) (2024)
- [What Makes and Breaks Safety Fine-tuning? A Mechanistic Study](https://mlsanity.com/paper/2407.10264/) (2024)
- [Misusing Tools in Large Language Models With Visual Adversarial Examples](https://mlsanity.com/paper/2310.03185/) (2023)
- [Understanding and Enhancing the Transferability of Adversarial Examples](https://mlsanity.com/paper/1802.09707/) (2018)
- [Understanding Jailbreak Success: A Study of Latent Space Dynamics in Large Language Models](https://mlsanity.com/paper/2406.09289/) (2024)
- [Red Teaming GPT-4V: Are GPT-4V Safe Against Uni/Multi-Modal Jailbreak Attacks?](https://mlsanity.com/paper/2404.03411/) (2024)
- [Reading Isn't Believing: Adversarial Attacks On Multi-Modal Neurons](https://mlsanity.com/paper/2103.10480/) (2021)
- [Exploring the Transferability of Visual Prompting for Multimodal Large Language Models](https://mlsanity.com/paper/2404.11207/) (2024)
- [Self-interpreting Adversarial Images](https://mlsanity.com/paper/2407.08970/) (2024)
- [Investigating Adversarial Trigger Transfer in Large Language Models](https://mlsanity.com/paper/2404.16020/) (2024)
- [Instance-Level Trojan Attacks on Visual Question Answering via Adversarial Learning in Neuron Activation Space](https://mlsanity.com/paper/2304.00436/) (2023)
- [Multi-attacks: Many images + the same adversarial attack → many target labels](https://mlsanity.com/paper/2308.03792/) (2023)

## Built on by (most cited first, 30 of 30)

- [Best-of-N Jailbreaking](https://mlsanity.com/paper/2412.03556/) (2024)
- [Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attacks](https://mlsanity.com/paper/2502.13175/) (2025)
- [Adversarial Attacks on Robotic Vision Language Action Models](https://mlsanity.com/paper/2506.03350/) (2025)
- [Jailbreak-AudioBench: In-Depth Evaluation and Analysis of Jailbreak Threats for Large Audio Language Models](https://mlsanity.com/paper/2501.13772/) (2025)
- [CaMeLs Can Use Computers Too: System-level Security for Computer Use Agents](https://mlsanity.com/paper/2601.09923/) (2026)
- [Gradient-based Jailbreak Images for Multimodal Fusion Models](https://mlsanity.com/paper/2410.03489/) (2024)
- [Speech-Audio Compositional Attacks on Multimodal LLMs and Their Mitigation with SALMONN-Guard](https://mlsanity.com/paper/2511.10222/) (2025)
- [Ensemble everything everywhere: Multi-scale aggregation for adversarial robustness](https://mlsanity.com/paper/2408.05446/) (2024)
- [MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agents](https://mlsanity.com/paper/2503.10809/) (2025)
- [Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025](https://mlsanity.com/paper/2506.12430/) (2025)
- [FORCE: Transferable Visual Jailbreaking Attacks via Feature Over-Reliance CorrEction](https://mlsanity.com/paper/2509.21029/) (2025)
- ["I am bad": Interpreting Stealthy, Universal and Robust Audio Jailbreaks in Audio-Language Models](https://mlsanity.com/paper/2502.00718/) (2025)
- [Jailbreak Defense in a Narrow Domain: Limitations of Existing Methods and a New Transcript-Classifier Approach](https://mlsanity.com/paper/2412.02159/) (2024)
- [Jailbreak Transferability Emerges from Shared Representations](https://mlsanity.com/paper/2506.12913/) (2025)
- [Simulated Ensemble Attack: Transferring Jailbreaks Across Fine-tuned Vision-Language Models](https://mlsanity.com/paper/2508.01741/) (2025)
- [Less Is More -- Until It Breaks: Security Pitfalls of Vision Token Compression in Large Vision-Language Models](https://mlsanity.com/paper/2601.12042/) (2026)
- [Toward Universal and Transferable Jailbreak Attacks on Vision-Language Models](https://mlsanity.com/paper/2602.01025/) (2026)
- [Disrupting Hierarchical Reasoning: Adversarial Protection for Geographic Privacy in Multimodal Reasoning Models](https://mlsanity.com/paper/2512.08503/) (2025)
- [Furina: Fragmented Uncertainty-Driven Refusal Instability Attack](https://mlsanity.com/paper/2605.26158/) (2026)
- [Visual Token Compression Enhances Robustness of MLLMs](https://mlsanity.com/paper/2607.22716/) (2026)
- [VISOR++: Universal Visual Inputs based Steering for Large Vision Language Models](https://mlsanity.com/paper/2509.25533/) (2025)
- [Understanding Adversarial Transfer: Why Representation-Space Attacks Fail Where Data-Space Attacks Succeed](https://mlsanity.com/paper/2510.01494/) (2025)
- [Why Does Weak-OOD Help? A Further Step Towards Understanding Jailbreaking VLMs](https://mlsanity.com/paper/2511.08367/) (2025)
- [The Persistent Vulnerability of Aligned AI Systems](https://mlsanity.com/paper/2604.00324/) (2026)
- [Jailbreaking Frontier Foundation Models Through Intention Deception](https://mlsanity.com/paper/2604.24082/) (2026)
- [Laundering AI Authority with Adversarial Examples](https://mlsanity.com/paper/2605.04261/) (2026)
- [Break the Brake, Not the Wheel: Untargeted Jailbreak via Entropy Maximization](https://mlsanity.com/paper/2605.10764/) (2026)
- [Overloading Large Vision-Language Models for Jailbreaking](https://mlsanity.com/paper/2607.02961/) (2026)
- [One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMs](https://mlsanity.com/paper/2609.33833/) (2026)
- [VEX-Bench: Benchmarking Verification Complexity of LLM-Generated Misinformation](https://mlsanity.com/paper/2609.35028/) (2026)

## Abstract (arXiv)

The integration of new modalities into frontier AI systems offers exciting capabilities, but also increases the possibility such systems can be adversarially manipulated in undesirable ways. In this work, we focus on a popular class of vision-language models (VLMs) that generate text outputs conditioned on visual and textual inputs. We conducted a large-scale empirical study to assess the transferability of gradient-based universal image ``jailbreaks" using a diverse set of over 40 open-parameter VLMs, including 18 new VLMs that we publicly release. Overall, we find that transferable gradient-based image jailbreaks are extremely difficult to obtain. When an image jailbreak is optimized against a single VLM or against an ensemble of VLMs, the jailbreak successfully jailbreaks the attacked VLM(s), but exhibits little-to-no transfer to any other VLMs; transfer is not affected by whether the attacked and target VLMs possess matching vision backbones or language models, whether the language model underwent instruction-following and/or safety-alignment training, or many other factors. Only two settings display partially successful transfer: between identically-pretrained and identically-initialized VLMs with slightly different VLM training data, and between different training checkpoints of a single VLM. Leveraging these results, we then demonstrate that transfer can be significantly improved against a specific target VLM by attacking larger ensembles of ``highly-similar" VLMs. These results stand in stark contrast to existing evidence of universal and transferable text jailbreaks against language models and transferable adversarial attacks against image classifiers, suggesting that VLMs may be more robust to gradient-based transfer attacks.