mlsanityData to 14 Oct 2026

Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast

arXiv 2402.08567 ↗First posted 13 Feb 2024cs.CL cs.CR cs.CVcited by 153

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 2 more
Abstract (arXiv)

A multimodal large language model (MLLM) agent can receive instructions, capture images, retrieve histories from memory, and decide which tools to use. Nonetheless, red-teaming efforts have revealed that adversarial images/prompts can jailbreak an MLLM and cause unaligned behaviors. In this work, we report an even more severe safety issue in multi-agent environments, referred to as infectious jailbreak. It entails the adversary simply jailbreaking a single agent, and without any further intervention from the adversary, (almost) all agents will become infected exponentially fast and exhibit harmful behaviors. To validate the feasibility of infectious jailbreak, we simulate multi-agent environments containing up to one million LLaVA-1.5 agents, and employ randomized pair-wise chat as a proof-of-concept instantiation for multi-agent interaction. Our results show that feeding an (infectious) adversarial image into the memory of any randomly chosen agent is sufficient to achieve infectious jailbreak. Finally, we derive a simple principle for determining whether a defense mechanism can provably restrain the spread of infectious jailbreak, but how to design a practical defense that meets this principle remains an open question to investigate. Our project page is available at https://sail-sg.github.io/Agent-Smith/.

Key facts. Cited by 153 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 82 of its 104 references that are in the map. First posted on arXiv 13 Feb 2024.

Builds on

82 of 104 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023How Robust is Google's Bard to Adversarial Image Attacks?21 Sep
  2. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul
  3. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun
  4. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
  5. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 77

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
38 more
  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  11. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  12. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  13. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  14. 2016Adversarial examples in the physical worldcited by 1,769
  15. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  16. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  17. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  18. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  19. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  20. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  21. 2022Red Teaming Language Models with Language Modelscited by 1,026
  22. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  23. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  24. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  25. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  26. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  27. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 599
  28. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 598
  29. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
  30. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
  31. 2017Adversarial Patchcited by 517
  32. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  33. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  34. 2022Diffusion Models for Adversarial Purificationcited by 476
  35. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  36. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
  37. 2023Cognitive Architectures for Language Agentscited by 429
  38. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 33; 2025: 53; 2026: 67.

153 papers.

Sort:

By area:

  1. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 205
  2. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systemscited by 133
  3. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 127
  4. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  5. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 125
  6. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 125
  7. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Modelscited by 120
  8. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 120
  9. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  10. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 110
  11. 2024On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agentscited by 99
  12. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 96
38 more
  1. 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communitiescited by 86
  2. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 84
  3. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 82
  4. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 80
  5. 2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 71
  6. 2024Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Charactercited by 68
  7. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 67
  8. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 65
  9. 2025G-Safeguard: A Topology-Guided Security Lens and Treatment on LLM-based Multi-agent Systemscited by 59
  10. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 54
  11. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 53
  12. 2024Machine Unlearning in Generative AI: A Surveycited by 49
  13. 2025SAGA: A Security Architecture for Governing AI Agentic Systemscited by 45
  14. 2025Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignmentcited by 43
  15. 2025Infrastructure for AI Agentscited by 41
  16. 2024Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Modelscited by 38
  17. 2024Core Knowledge Deficits in Multi-Modal Language Modelscited by 38
  18. 2025AgentSafe: Safeguarding Large Language Model-based Multi-agent Systems via Hierarchical Data Managementcited by 38
  19. 2025textitAgents Under Siege: Breaking Pragmatic Multi-Agent LLM Systems with Optimized Prompt Attackscited by 34
  20. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 32
  21. 2024NetSafe: Exploring the Topological Safety of Multi-agent Networkscited by 29
  22. 2026A Survey on Long-Term Memory Security in LLM Agents: Attacks, Defenses, and Governance Across the Memory Lifecyclecited by 27
  23. 2026"Humans welcome to observe": A First Look at the Agent Social Network Moltbookcited by 26
  24. 2024From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreakingcited by 24
  25. 2026Poison Once, Exploit Forever: Environment-Injected Memory Poisoning Attacks on Web Agentscited by 24
  26. 2024Adversarial Attacks of Vision Tasks in the Past 10 Years: A Surveycited by 23
  27. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  28. 2025LLM-Powered AI Agent Systems and Their Applications in Industrycited by 22
  29. 2024The Wolf Within: Covert Injection of Malice into MLLM Societies via an MLLM Operativecited by 21
  30. 2024Granular Privacy Control for Geolocation with Vision Language Modelscited by 21
  31. 2024The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defensecited by 21
  32. 2025CORBA: Contagious Recursive Blocking Attacks on Multi-Agent Systems Based on Large Language Modelscited by 20
  33. 2024Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Securitycited by 19
  34. 2024RapGuard: Safeguarding Multimodal Large Language Models via Rationale-aware Defensive Promptingcited by 19
  35. 2024VLMGuard: Bootstrapping Malicious Prompt Detectors from Unlabeled Vision-Language Prompts in the Wildcited by 18
  36. 2026Colosseum: Auditing Collusion in Cooperative Multi-Agent Systemscited by 18
  37. 2025Survey of Adversarial Robustness in Multimodal Large Language Modelscited by 16
  38. 2026SoK: The Attack Surface of Agentic AI - Tools and Autonomycited by 16

50 most cited of 153 All 153 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Multi-Agent Defences Against Jailbreaking Attacks on Large Language Models
  2. 2025JPS: Jailbreak Multimodal Large Language Models with Collaborative Visual Perturbation and Textual Steering
  3. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks
  4. 2025Defending Large Language Models Against Jailbreak Exploits with Responsible AI Considerations
  5. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Models
  6. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet
  7. 2025AutoAdv: Automated Adversarial Prompting for Multi-Turn Jailbreaking of Large Language Models
  8. 2025AutoAdv: Automated Adversarial Prompting for Multi-Turn Jailbreaking of Large Language Models