mlsanityData to 14 Oct 2026

Implicit Jailbreak Attacks via Cross-Modal Information Concealment on Vision-Language Models

arXiv 2505.16446 ↗First posted 22 May 2025cs.LGcited by 6

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more
Abstract (arXiv)

Multimodal large language models (MLLMs) enable powerful cross-modal reasoning capabilities. However, the expanded input space introduces new attack surfaces. Previous jailbreak attacks often inject malicious instructions from text into less aligned modalities, such as vision. As MLLMs increasingly incorporate cross-modal consistency and alignment mechanisms, such explicit attacks become easier to detect and block. In this work, we propose a novel implicit jailbreak framework termed IJA that stealthily embeds malicious instructions into images via least significant bit steganography and couples them with seemingly benign, image-related textual prompts. To further enhance attack effectiveness across diverse MLLMs, we incorporate adversarial suffixes generated by a surrogate model and introduce a template optimization module that iteratively refines both the prompt and embedding based on model feedback. On commercial models like GPT-4o and Gemini-1.5 Pro, our method achieves attack success rates of over 90% using an average of only 3 queries.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 57 of its 62 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

57 of 62 references.

References · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2025Qwen2.5-VL Technical Reportcited by 5,587
  9. 2013Intriguing properties of neural networkscited by 5,155
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  12. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  8. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  9. 2020Extracting Training Data from Large Language Modelscited by 2,189
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  11. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  12. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,106
  13. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  14. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  15. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
  16. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
  17. 2023Are aligned neural networks adversarially aligned?cited by 315
  18. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  19. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  20. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  21. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 273
  22. 2023Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Modelscited by 175
  23. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 172
  24. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Modelscited by 136
  25. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 135
  26. 2024Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversationscited by 129
  27. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 125
  28. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learningcited by 115
  29. 2023On the Robustness of Large Multimodal Models Against Image Adversarial Attackscited by 92
  30. 2024CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Modelscited by 76
  31. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 67
  32. 2024VLSBench: Unveiling Visual Leakage in Multimodal Safetycited by 63
  33. 2024Jailbreak Large Vision-Language Models Through Multi-Modal Linkagecited by 62
  34. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Imagecited by 57
  35. 2024ShieldLM: Empowering LLMs as Aligned, Customizable and Explainable Safety Detectorscited by 51
  36. 2025Jailbreaking Multimodal Large Language Models via Shuffle Inconsistencycited by 51
  37. 2024Arondight: Red Teaming Large Vision Language Models with Auto-generated Multi-modal Jailbreak Promptscited by 42
  38. 2025Distraction is All You Need for Multimodal Large Language Model Jailbreakingcited by 36

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025OmniSafeBench-MM: A Unified Benchmark and Toolbox for Multimodal Jailbreak Attack-Defense Evaluationcited by 11
  2. 2026SafeNeuron: Neuron-Level Safety Alignment for Large Language Modelscited by 10
  3. 2026Multilingual Safety Alignment Via Sparse Weight Editingcited by 6
  4. 2025GuardTrace-VL: Detecting Unsafe Multimodel Reasoning via Iterative Safety Supervisioncited by 4
  5. 2026TreeTeaming: Autonomous Red-Teaming of Vision-Language Models via Hierarchical Strategy Explorationcited by 3
  6. 2026Attack Ensembles Expose a Safety-Utility Trade-off in Black-Box Guard Defenses Against Encoded VLM Jailbreakscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Cross-Modal Obfuscation for Jailbreak Attacks on Large Vision-Language Models
  2. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt
  3. 2026Overloading Large Vision-Language Models for Jailbreaking
  4. 2024Cross-modality Information Check for Detecting Jailbreaking in Multimodal Large Language Models
  5. 2025DefenSee: Dissecting Threat from Sight and Text -- A Multi-View Defensive Pipeline for Multi-modal Jailbreaks
  6. 2025Align is not Enough: Multimodal Universal Jailbreak Attack against Multimodal Large Language Models
  7. 2025Odysseus: Jailbreaking Commercial Multimodal LLM-integrated Systems via Dual Steganography
  8. 2025PolyJailbreak: Cross-Modal Jailbreaking Attacks on Black-Box Multimodal LLMs