mlsanityData to 7 Oct 2026

Efficient LLM-Jailbreaking via Multimodal-LLM Jailbreak

arXiv 2405.20015 ↗First posted 30 May 2024cs.AI cs.CLcited by 4

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

This paper focuses on jailbreaking attacks against large language models (LLMs), eliciting them to generate objectionable content in response to harmful user queries. Unlike previous LLM-jailbreak methods that directly orient to LLMs, our approach begins by constructing a multimodal large language model (MLLM) built upon the target LLM. Subsequently, we perform an efficient MLLM jailbreak and obtain a jailbreaking embedding. Finally, we convert the embedding into a textual jailbreaking suffix to carry out the jailbreak of target LLM. Compared to the direct LLM-jailbreak methods, our indirect jailbreaking approach is more efficient, as MLLMs are more vulnerable to jailbreak than pure LLM. Additionally, to improve the attack success rate of jailbreak, we propose an image-text semantic matching scheme to identify a suitable initial input. Extensive experiments demonstrate that our approach surpasses current state-of-the-art jailbreak methods in terms of both efficiency and effectiveness. Moreover, our approach exhibits superior cross-class generalization abilities.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 17 of its 31 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

17 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 15

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  10. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  12. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
3 more
  1. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  2. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  3. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 1.

4 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks
  2. 2024Jailbreaking Attack against Multimodal Large Language Model
  3. 2024From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreaking
  4. 2024Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Models
  5. 2024textitMMJ-Bench: A Comprehensive Study on Jailbreak Attacks and Defenses for Multimodal Large Language Models
  6. 2025Probabilistic Modeling of Jailbreak on Multimodal LLMs: From Quantification to Application
  7. 2026DMN: A Compositional Framework for Jailbreaking Multimodal LLMs with Multi-Image Inputs
  8. 2025Align is not Enough: Multimodal Universal Jailbreak Attack against Multimodal Large Language Models