mlsanityData to 14 Oct 2026

Every Picture Tells a Dangerous Story: Memory-Augmented Multi-Agent Jailbreak Attacks on VLMs

arXiv 2604.12616 ↗First posted 14 Apr 2026cs.AI cs.MMcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 2 more
Abstract (arXiv)

Vision-Language Models (VLMs) expand the attack surface of safety-aligned systems by coupling visual perception with text generation. Existing multimodal jailbreak attacks primarily rely on crafted visual content, adversarial perturbations, or image-specific attack strategies, leaving the potential of reusable visual anchors in benign natural images largely unexplored. To address the problem, we introduce MemJack, a memory-augmented multi-agent framework for automated VLM red-teaming with natural images. MemJack combines visual-anchor discovery, visual-semantic camouflage, response evaluation, reflection-guided repair, and dynamic replanning within a closed-loop attack pipeline. Beyond attack generation, MemJack automatically transforms public images into attack anchors, enables the construction of MemJack-Bench, a dataset of over 113,000 interactive multimodal jailbreak trajectories for safety evaluation and defensive alignment. Extensive empirical evaluations across full, unmodified COCO val2017 images demonstrate that MemJack achieves a 71.48% attack success rate (ASR) against Qwen3-VL-Plus, scaling to 90% under extended budgets. Compared with representative multimodal jailbreak baselines under the same natural-image evaluation setting, MemJack achieves the highest ASR while requiring fewer mean rounds to success, demonstrating superior jailbreak effectiveness on VLMs. These results demonstrate that benign natural images can act as transferable jailbreak anchors and reveal substantial vulnerabilities in current safety-aligned VLMs.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 31 references that are in the map. First posted on arXiv 14 Apr 2026.

Builds on

26 of 31 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026Red-teaming the Multimodal Reasoning: Jailbreaking Vision-Language Models via Cross-modal Entanglement Attacks9 Feb
  2. 2025Visual Contextual Attack: Jailbreaking MLLMs with Image-Driven Context Injection3 Jul
  3. 2025JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Models26 May
  4. 2024IDEATOR: Jailbreaking and Benchmarking Large Vision-Language Models Using Themselves29 Oct
  5. 2024Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Model21 Jun
  6. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks3 Apr
  7. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models14 Mar
  8. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models29 Nov
  9. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  10. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun

Other references · 16

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  3. 2017Billion-scale similarity search with GPUscited by 2,497
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  6. 2022Red Teaming Language Models with Language Modelscited by 1,026
  7. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  8. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 455
  9. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 429
  10. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 230
  11. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 199
  12. 2025Qwen3Guard Technical Reportcited by 155
4 more
  1. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 135
  2. 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 102
  3. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 97
  4. 2025Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategycited by 44

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Evolving Skill-Structured Attack Memory Enhances LLM Jailbreakingcited by 3
  2. 2026Automatic Hard Example Synthesis with Multi-Level Agentic Data Curationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attacks
  2. 2026Fully Unleashing the Multimodal Attacker: Meta-Adaptive Jailbreaking of Vision-Language Models
  3. 2025Tit-for-Tat: Safeguarding Large Vision-Language Models Against Jailbreak Attacks via Adversarial Defense
  4. 2026Jailbreaking Vision-Language Models Through the Visual Modality
  5. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Image
  6. 2026Beyond Visual Safety: Jailbreaking Multimodal Large Language Models for Harmful Image Generation via Semantic-Agnostic Inputs
  7. 2026When the Prompt Becomes Visual: Vision-Centric Jailbreak Attacks for Large Image Editing Models
  8. 2025TAIJI: Textual Anchoring for Immunizing Jailbreak Images in Vision Language Models