mlsanityData to 14 Oct 2026

AgentHijack: Visual Patch Attacks on Multimodal Computer-Use Agents

arXiv 2609.09212 ↗First posted 6 Sep 2026cs.CR cs.AI cs.CYcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

This paper presents an end-to-end evaluation framework for image-triggered command injection against computer-use agents (CUAs). The goal is to test whether a local visual patch can induce verifiable environmental consequences along the full chain of screenshot input, VLM generation, action parsing, and environment execution. We train and deploy patches on author-controlled GitHub Pages pages and a locally deployed CSDN clone, and evaluate them in real environments across five open-source or publicly available GUI-agent or vision-language-model (VLM) backends. Our experiment aggregates 600 instance-level online cases, with T-ASR, TAPR, and E2E-ASR reaching 84.5%, 47.0%, and 20.3%, respectively. Trajectory analysis further shows that in some successful cases the agent first executes a malicious terminal command and then continues the original benign task. These results indicate that optimized local visual signals can affect not only VLM outputs but also propagate through the execution pipeline of open CUAs and create real environmental risk.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 42 of its 44 references that are in the map. First posted on arXiv 6 Sep 2026.

Builds on

42 of 44 references.

Closest ancestors · 33

Same direction, newest first

  1. 2026"What Did It Actually Do?": Understanding Risk Awareness and Traceability for Computer-Use Agents30 Mar
  2. 2025HackWorld: Evaluating Computer-Use Agents on Exploiting Web Application Vulnerabilities14 Oct
  3. 2025SecureWebArena: A Holistic Security Evaluation Benchmark for LVLM-based Web Agents11 Oct
  4. 2025WAInjectBench: Benchmarking Prompt Injection Detections for Web Agents1 Oct
  5. 2025Secure and Efficient Access Control for Computer-Use Agents via Context Space26 Sep
  6. 2025AgentSentinel: An End-to-End and Real-Time Security Defense Framework for Computer-Use Agents9 Sep
  7. 2025Manipulating LLM Web Agents with Indirect Prompt Injection Attack via HTML Accessibility Tree20 Jul
  8. 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows29 Jun
  9. 2025OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents17 Jun
  10. 2025Poison Once, Control Anywhere: Clean-Text Visual Backdoors in VLM-based Mobile Agents16 Jun
  11. 2025VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents3 Jun
  12. 2025MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environments2 Jun
21 more
  1. 2025AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agents31 May
  2. 2025RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agents31 May
  3. 2025LLM Agents Should Employ Security Principles29 May
  4. 2025RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environments28 May
  5. 2025AdInject: Real-World Black-Box Attacks on Web Agents via Advertising Delivery27 May
  6. 2025Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM-Powered Mobile GUI Agents20 May
  7. 2025WebInject: Prompt Injection Attack to Web Agents16 May
  8. 2025AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents9 May
  9. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injection19 Apr
  10. 2025The Obvious Invisible Threat: LLM-Powered GUI Agents' Vulnerability to Fine-Print Injections15 Apr
  11. 2025MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agents13 Mar
  12. 2025SafeArena: Evaluating the Safety of Autonomous Web Agents6 Mar
  13. 2025Evaluating the Robustness of Multimodal Agents Against Active Environmental Injection Attacks18 Feb
  14. 2024Attacking Vision-Language Computer Agents via Pop-ups4 Nov
  15. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents3 Oct
  16. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage17 Sep
  17. 2024Dissecting Adversarial Robustness of Multimodal LM Agents18 Jun
  18. 2024BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents5 Jun
  19. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  20. 2024WIPI: A New Web Threat for LLM-Driven Web Agents26 Feb
  21. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents17 Feb

Other references · 9

  1. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  2. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  3. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
  4. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
  5. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  6. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 160
  7. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 114
  8. 2024Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agentscited by 56
  9. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 56

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MMSkillRisk: Can Agents Stay Safe When Multimodal Skills Become Traps?
  2. 2025A Systematization of Security Vulnerabilities in Computer Use Agents
  3. 2026(A)I Sees What You Don't: Exploiting New Attack Surfaces in Third-Party Mobile Agents
  4. 2026VisualLeakBench: Reproducible Action-Boundary Propagation Failures in Vision-Language Agents
  5. 2025VisualTrap: A Stealthy Backdoor Attack on GUI Agents via Visual Grounding Manipulation
  6. 2026Visual Confused Deputy: Exploiting and Defending Perception Failures in Computer-Using Agents
  7. 2026SnapGuard: Lightweight Prompt Injection Detection for Screenshot-Based Web Agents
  8. 2026Domain-Conditioned Safety in Frontier Computer-Using Agents: A 793-Episode Browser Benchmark, a Coding-Domain Cross-Reference, and a Reproducibility Audit of Recent Red-Teaming