mlsanityData to 7 Oct 2026

Dissecting Adversarial Robustness of Multimodal LM Agents

arXiv 2406.12814 ↗First posted 18 Jun 2024cs.LG cs.CL cs.CRcited by 118

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

As language models (LMs) are used to build autonomous agents in real environments, ensuring their adversarial robustness becomes a critical challenge. Unlike chatbots, agents are compound systems with multiple components taking actions, which existing LMs safety evaluations do not adequately address. To bridge this gap, we manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena, a real environment for web agents. To systematically examine the robustness of agents, we propose the Agent Robustness Evaluation (ARE) framework. ARE views the agent as a graph showing the flow of intermediate outputs between components and decomposes robustness as the flow of adversarial information on the graph. We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search. With imperceptible perturbations to a single image (less than 5% of total web page pixels), an attacker can hijack these agents to execute targeted adversarial goals with success rates up to 67%. We also use ARE to rigorously evaluate how the robustness changes as new components are added. We find that inference-time compute that typically improves benign performance can open up new vulnerabilities and harm robustness. An attacker can compromise the evaluator used by the reflexion agent and the value function of the tree search agent, which increases the attack success relatively by 15% and 20%. Our data and code for attacks, defenses, and evaluation are at https://github.com/ChenWu98/agent-attack

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 61 of its 70 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

61 of 70 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024A Trembling House of Cards? Mapping Adversarial Attacks against Language Agents15 Feb
  2. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast13 Feb
  3. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 58

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  9. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  11. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  12. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
38 more
  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  5. 2016Adversarial examples in the physical worldcited by 1,762
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  7. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  8. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  10. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  11. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  12. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  13. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  14. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  15. 2016Adversarial Machine Learning at Scalecited by 934
  16. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  17. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  18. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  19. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  20. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  21. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  22. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  23. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  24. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  25. 2023Language Models can Solve Computer Taskscited by 371
  26. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  27. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  28. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 325
  29. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  30. 2018Certified Defenses against Adversarial Examplescited by 305
  31. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  32. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 260
  33. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  34. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 233
  35. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  36. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  37. 2023How Robust is Google's Bard to Adversarial Image Attacks?cited by 177
  38. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 165

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 67; 2026: 43.

118 papers.

Sort:

By area:

  1. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  2. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 151
  3. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 145
  4. 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoningcited by 144
  5. 2025OS-Harm: A Benchmark for Measuring Safety of Computer Use Agentscited by 104
  6. 2025SafeArena: Evaluating the Safety of Autonomous Web Agentscited by 97
  7. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  8. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 82
  9. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 71
  10. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  11. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 61
  12. 2024AdvAgent: Controllable Blackbox Red-teaming on Web Agentscited by 57
38 more
  1. 2025The Obvious Invisible Threat: LLM-Powered GUI Agents' Vulnerability to Fine-Print Injectionscited by 55
  2. 2025WebInject: Prompt Injection Attack to Web Agentscited by 48
  3. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 43
  4. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injectioncited by 42
  5. 2025AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agentscited by 42
  6. 2025RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environmentscited by 42
  7. 2025VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agentscited by 42
  8. 2025BadVLA: Towards Backdoor Attacks on Vision-Language-Action Models via Objective-Decoupled Optimizationcited by 37
  9. 2024AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Drivingcited by 35
  10. 2025RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agentscited by 34
  11. 2025DoomArena: A framework for Testing AI Agents Against Evolving Security Threatscited by 31
  12. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 28
  13. 2025Adversarial Reasoning at Jailbreaking Timecited by 27
  14. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  15. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  16. 2025MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environmentscited by 23
  17. 2025SAFEFLOW: A Principled Protocol for Trustworthy and Transactional Autonomous Agent Systemscited by 23
  18. 2025BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agentscited by 23
  19. 2025Evaluating the Robustness of Multimodal Agents Against Active Environmental Injection Attackscited by 21
  20. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  21. 2025WAInjectBench: Benchmarking Prompt Injection Detections for Web Agentscited by 21
  22. 2025WebGuard: Building a Generalizable Guardrail for Web Agentscited by 20
  23. 2026MUZZLE: Adaptive Agentic Red-Teaming of Web Agents Against Indirect Prompt Injection Attackscited by 18
  24. 2026Poison Once, Exploit Forever: Environment-Injected Memory Poisoning Attacks on Web Agentscited by 18
  25. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  26. 2025Survey of Adversarial Robustness in Multimodal Large Language Modelscited by 14
  27. 2025On the Robustness of GUI Grounding Models Against Image Attackscited by 14
  28. 2025Think Twice Before You Act: Enhancing Agent Behavioral Safety with Thought Correctioncited by 14
  29. 2025DECEPTICON: How Dark Patterns Manipulate Web Agentscited by 13
  30. 2025Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generationcited by 12
  31. 2025MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agentscited by 12
  32. 2025Aligning Multimodal LLM with Human Preference: A Surveycited by 12
  33. 2025GhostEI-Bench: Do Mobile Agents Resilience to Environmental Injection in Dynamic On-Device Environments?cited by 12
  34. 2026AdapTools: Adaptive Tool-based Indirect Prompt Injection Attacks on Agentic LLMscited by 12
  35. 2026The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agentscited by 12
  36. 2025Stop Wasting Your Tokens: Towards Efficient Runtime Multi-Agent Systemscited by 11
  37. 2026SafePred: A Predictive Guardrail for Computer-Using Agents via World Modelscited by 11
  38. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Modelscited by 10

50 most cited of 118 All 118 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systems
  2. 2025Benchmarking the Robustness of Agentic Systems to Adversarially-Induced Harms
  3. 2025Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectives
  4. 2026ARB4WM: An Adversarial Robustness Benchmark for World Models in Continuous Control
  5. 2026OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs
  6. 2026Agentic Adversarial Rewriting Exposes Architectural Vulnerabilities in Black-Box NLP Pipelines
  7. 2026Dual-Modality Multi-Stage Adversarial Safety Training: Robustifying Multimodal Web Agents Against Cross-Modal Attacks
  8. 2026GuardianAgentBench: Where Agents Fail and How to Guard Them