mlsanityData to 7 Oct 2026

Jailbreak Large Vision-Language Models Through Multi-Modal Linkage

arXiv 2412.00473 ↗First posted 30 Nov 2024cs.CVcited by 53

On the map: LLM safety and security → Vision-language and text-to-image safety

Abstract (arXiv)

With the significant advancement of Large Vision-Language Models (VLMs), concerns about their potential misuse and abuse have grown rapidly. Previous studies have highlighted VLMs' vulnerability to jailbreak attacks, where carefully crafted inputs can lead the model to produce content that violates ethical and legal standards. However, existing methods struggle against state-of-the-art VLMs like GPT-4o, due to the over-exposure of harmful content and lack of stealthy malicious guidance. In this work, we propose a novel jailbreak attack framework: Multi-Modal Linkage (MML) Attack. Drawing inspiration from cryptography, MML utilizes an encryption-decryption process across text and image modalities to mitigate over-exposure of malicious information. To align the model's output with malicious intent covertly, MML employs a technique called "evil alignment", framing the attack within a video game production scenario. Comprehensive experiments demonstrate MML's effectiveness. Specifically, MML jailbreaks GPT-4o with attack success rates of 97.80% on SafeBench, 98.81% on MM-SafeBench and 99.07% on HADES-Dataset. Our code is available at https://github.com/wangyu-ovo/MML.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in Vision-language and text-to-image safety, one of 12 directions in LLM safety and security. Builds on 27 of its 48 references that are in the map. First posted on arXiv 30 Nov 2024.

Builds on

27 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting14 Mar
  2. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  3. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul

Other references · 24

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  10. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  11. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  12. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
12 more
  1. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  2. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  3. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  4. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  5. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attackscited by 208
  6. 2023How Robust is Google's Bard to Adversarial Image Attacks?cited by 177
  7. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  8. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  9. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111
  10. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  11. 2024Fight Back Against Jailbreaking via Prompt Adversarial Tuningcited by 63
  12. 2024When "Competency" in Reasoning Opens the Door to Vulnerability: Jailbreaking LLMs via Novel Complex Cipherscited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 27; 2026: 26.

53 papers.

Sort:

By area:

  1. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  2. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  3. 2025Backdoor Cleaning without External Guidance in MLLM Fine-tuningcited by 34
  4. 2025SafeMLRM: Demystifying Safety in Multi-modal Large Reasoning Modelscited by 29
  5. 2025FC-Attack: Jailbreaking Multimodal Large Language Models via Auto-Generated Flowchartscited by 15
  6. 2025Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025cited by 11
  7. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 11
  8. 2025SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimizationcited by 11
  9. 2025USB: A Comprehensive and Unified Safety Evaluation Benchmark for Multimodal Large Language Modelscited by 10
  10. 2025JPS: Jailbreak Multimodal Large Language Models with Collaborative Visual Perturbation and Textual Steeringcited by 10
  11. 2025OmniSafeBench-MM: A Unified Benchmark and Toolbox for Multimodal Jailbreak Attack-Defense Evaluationcited by 10
  12. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 9
38 more
  1. 2025EVA: Evolving Semantic Adversaries for Red-Teaming GUI Agents Against Environmental Injection Attackscited by 8
  2. 2025Speech-Audio Compositional Attacks on Multimodal LLMs and Their Mitigation with SALMONN-Guardcited by 8
  3. 2025Implicit Jailbreak Attacks via Cross-Modal Information Concealment on Vision-Language Modelscited by 5
  4. 2025OMNIGUARD: An Efficient Approach for AI Safety Moderation Across Languages and Modalitiescited by 5
  5. 2025On Surjectivity of Neural Networks: Can you elicit any behavior from your model?cited by 4
  6. 2026Narrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agentscited by 4
  7. 2026Visual Exclusivity Attacks: Automatic Multimodal Red Teaming via Agentic Planningcited by 4
  8. 2026Understanding and Defending VLM Jailbreaks via Jailbreak-Related Representation Shiftcited by 3
  9. 2026Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Surveycited by 3
  10. 2026Jailbreaking Vision-Language Models Through the Visual Modalitycited by 3
  11. 2025Probabilistic Modeling of Jailbreak on Multimodal LLMs: From Quantification to Applicationcited by 2
  12. 2025From Hallucinations to Jailbreaks: Rethinking the Vulnerability of Large Foundation Modelscited by 2
  13. 2025Behind the Mask: Benchmarking Camouflaged Jailbreaks in Large Language Modelscited by 2
  14. 2025JaiLIP: Jailbreaking Vision-Language Models via Loss Guided Image Perturbationcited by 2
  15. 2026When the Prompt Becomes Visual: Vision-Centric Jailbreak Attacks for Large Image Editing Modelscited by 2
  16. 2026TreeTeaming: Autonomous Red-Teaming of Vision-Language Models via Hierarchical Strategy Explorationcited by 2
  17. 2026Safe responses matter: Output-aware safety guardrail mitigate over-refusal in MLLMscited by 2
  18. 2025The Tower of Babel Revisited: Multilingual Jailbreak Prompts on Closed-Source Large Language Modelscited by 1
  19. 2025LLaVAShield: Safeguarding Multimodal Multi-Turn Dialogues in Vision-Language Modelscited by 1
  20. 2025Learning to Detect Unknown Jailbreak Attacks in Large Vision-Language Modelscited by 1
  21. 2025VERA-V: Variational Inference Framework for Jailbreaking Vision-Language Modelscited by 1
  22. 2025Jailbreaking Large Vision Language Models in Intelligent Transportation Systemscited by 1
  23. 2025Think-Reflect-Revise: A Policy-Guided Reflective Framework for Safety Alignment in Large Vision Language Modelscited by 1
  24. 2026Robustness of Vision Language Models Against Split-Image Harmful Input Attackscited by 1
  25. 2026MIDAS: Multi-Image Dispersion and Semantic Reconstruction for Jailbreaking MLLMscited by 1
  26. 2026Furina: Fragmented Uncertainty-Driven Refusal Instability Attackcited by 1
  27. 2026Multi-Turn Adaptive Prompting Attack on Large Vision-Language Modelscited by 0
  28. 2026Co-Evolutionary Multi-Modal Alignment via Structured Adversarial Evolutioncited by 0
  29. 2026Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMscited by 0
  30. 2026Mosaic: Multimodal Jailbreak against Closed-Source VLMs via Multi-View Ensemble Optimizationcited by 0
  31. 2026Conceal, Reconstruct, Jailbreak: Exploiting the Reconstruction-Concealment Tradeoff in MLLMscited by 0
  32. 2026DMN: A Compositional Framework for Jailbreaking Multimodal LLMs with Multi-Image Inputscited by 0
  33. 2026Jailbreaking Multimodal Large Language Models using Multi-Clip Videocited by 0
  34. 2026MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Modelscited by 0
  35. 2026PHANTOM: A Large-Scale Dataset of Multimodal Adversarial Attacks for Vision-Language Modelscited by 0
  36. 2026COMIC: Reference-Aware Safety Gating for Multimodal Large Language Modelscited by 0
  37. 2026TempJail: Temporal Jailbreak Attack against Large Vision-Language Models via Subtitle Schedulingcited by 0
  38. 2026ReFrame: Evidence-Guided Test-Time Safety Alignment in Multimodal Large Language Modelscited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Jailbreaking Attack against Multimodal Large Language Model
  2. 2024Red Teaming GPT-4V: Are GPT-4V Safe Against Uni/Multi-Modal Jailbreak Attacks?
  3. 2026Overloading Large Vision-Language Models for Jailbreaking
  4. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Image
  5. 2025PolyJailbreak: Cross-Modal Jailbreaking Attacks on Black-Box Multimodal LLMs
  6. 2025Align is not Enough: Multimodal Universal Jailbreak Attack against Multimodal Large Language Models
  7. 2024Image-to-Text Logic Jailbreak: Your Imagination can Help You Do Anything
  8. 2024Cross-modality Information Check for Detecting Jailbreaking in Multimodal Large Language Models