mlsanityData to 14 Oct 2026

ImgTrojan: Jailbreaking Vision-Language Models with ONE Image

arXiv 2403.02910 ↗First posted 5 Mar 2024cs.CV cs.AIcited by 57

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 2 more
Abstract (arXiv)

There has been an increasing interest in the alignment of large language models (LLMs) with human values. However, the safety issues of their integration with a vision module, or vision language models (VLMs), remain relatively underexplored. In this paper, we propose a novel jailbreaking attack against VLMs, aiming to bypass their safety barrier when a user inputs harmful instructions. A scenario where our poisoned (image, text) data pairs are included in the training data is assumed. By replacing the original textual captions with malicious jailbreak prompts, our method can perform jailbreak attacks with the poisoned images. Moreover, we analyze the effect of poison ratios and positions of trainable parameters on our attack's success rate. For evaluation, we design two metrics to quantify the success rate and the stealthiness of our attack. Together with a list of curated harmful instructions, a benchmark for measuring attack efficacy is provided. We demonstrate the efficacy of our attack by comparing it with baseline methods.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 39 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

20 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul

Other references · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  10. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  12. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
7 more
  1. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  2. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  3. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  4. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agentscited by 137
  5. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  6. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 108
  7. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 27; 2026: 15.

57 papers.

Sort:

By area:

  1. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Modelscited by 136
  2. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 109
  3. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 84
  4. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 67
  5. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 67
  6. 2024VHELM: A Holistic Evaluation of Vision Language Modelscited by 53
  7. 2024Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shiftcited by 49
  8. 2025Distraction is All You Need for Multimodal Large Language Model Jailbreakingcited by 36
  9. 2024Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Modelscited by 33
  10. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 32
  11. 2024Failures to Find Transferable Image Jailbreaks Between Vision-Language Modelscited by 30
  12. 2025Membership Inference Attacks Against Vision-Language Modelscited by 29
38 more
  1. 2025X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIPcited by 21
  2. 2024Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Securitycited by 19
  3. 2025Survey of Adversarial Robustness in Multimodal Large Language Modelscited by 16
  4. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  5. 2025Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Modelscited by 14
  6. 2025Speech-Audio Compositional Attacks on Multimodal LLMs and Their Mitigation with SALMONN-Guardcited by 13
  7. 2024Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Modelscited by 12
  8. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 12
  9. 2024Safe + Safe = Unsafe? Exploring How Safe Images Can Be Exploited to Jailbreak Large Vision-Language Modelscited by 10
  10. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflictcited by 10
  11. 2025SRD: Reinforcement-Learned Semantic Perturbation for Backdoor Defense in VLMscited by 9
  12. 2025Attacker's Noise Can Manipulate Your Audio-based LLM in the Real Worldcited by 9
  13. 2026Unveiling the Fragility of Vision-Language Models: Multi-Modal Adversarial Synergy via Texture-Constrained Perturbations and Cross-Modal Optimizationcited by 9
  14. 2024BaThe: Defense against the Jailbreak Attack in Multimodal Large Language Models by Treating Harmful Instruction as Backdoor Triggercited by 8
  15. 2025AutoAdv: Automated Adversarial Prompting for Multi-Turn Jailbreaking of Large Language Modelscited by 8
  16. 2024Zer0-Jack: A Memory-efficient Gradient-based Jailbreaking Method for Black-box Multi-modal Large Language Modelscited by 7
  17. 2025Con Instruction: Universal Jailbreaking of Multimodal Large Language Models via Non-Textual Modalitiescited by 7
  18. 2025Implicit Jailbreak Attacks via Cross-Modal Information Concealment on Vision-Language Modelscited by 6
  19. 2026JailWAM: Jailbreaking World Action Models in Robot Controlcited by 6
  20. 2025REVEAL: Multi-turn Evaluation of Image-Input Harms for Vision LLMcited by 5
  21. 2025Making Every Step Effective: Jailbreaking Large Vision-Language Models Through Hierarchical KV Equalizationcited by 4
  22. 2025CrossGuard: Safeguarding MLLMs against Joint-Modal Implicit Malicious Attackscited by 4
  23. 2025JPRO: Automated Multimodal Jailbreaking via Multi-Agent Collaboration Frameworkcited by 4
  24. 2026Tone Matters: The Impact of Linguistic Tone on Hallucination in VLMscited by 4
  25. 2026Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Surveycited by 4
  26. 2026Jailbreaking LLMs & VLMs: Mechanisms, Evaluation, and Unified Defensecited by 3
  27. 2026LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Modelscited by 3
  28. 2025Tit-for-Tat: Safeguarding Large Vision-Language Models Against Jailbreak Attacks via Adversarial Defensecited by 2
  29. 2025Trojan Horse Prompting: Jailbreaking Conversational Multimodal Models by Forging Assistant Messagecited by 2
  30. 2026Steal the Patch Size: Adversarially Manipulate Vision-Language Modelscited by 2
  31. 2024Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Modelscited by 1
  32. 2025Goal2Story: A Multi-Agent Fleet based on Privately Enabled sLLMs for Impacting Mapping on Requirements Elicitationcited by 1
  33. 2025Robustness Evaluation of OCR-based Visual Document Understanding under Multi-Modal Adversarial Attackscited by 1
  34. 2025VERA-V: Variational Inference Framework for Jailbreaking Vision-Language Modelscited by 1
  35. 2025Why Does Weak-OOD Help? A Further Step Towards Understanding Jailbreaking VLMscited by 1
  36. 2025Enhanced MLLM Black-Box Jailbreaking Attacks and Defensescited by 0
  37. 2025AutoAdv: Automated Adversarial Prompting for Multi-Turn Jailbreaking of Large Language Modelscited by 0
  38. 2026Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMscited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Models
  2. 2026Fully Unleashing the Multimodal Attacker: Meta-Adaptive Jailbreaking of Vision-Language Models
  3. 2026Jailbreaking Vision-Language Models Through the Visual Modality
  4. 2025JaiLIP: Jailbreaking Vision-Language Models via Loss Guided Image Perturbation
  5. 2026Robustness of Vision Language Models Against Split-Image Harmful Input Attacks
  6. 2026Toward Universal and Transferable Jailbreak Attacks on Vision-Language Models
  7. 2026Overloading Large Vision-Language Models for Jailbreaking
  8. 2025Learning to Detect Unseen Jailbreak Attacks in Large Vision-Language Models