mlsanityData to 7 Oct 2026

Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks

arXiv 2502.08586 ↗First posted 12 Feb 2025cs.LG cs.AIcited by 49

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

A high volume of recent ML security literature focuses on attacks against aligned large language models (LLMs). These attacks may extract private information or coerce the model into producing harmful outputs. In real-world deployments, LLMs are often part of a larger agentic pipeline including memory systems, retrieval, web access, and API calling. Such additional components introduce vulnerabilities that make these LLM-powered agents much easier to attack than isolated LLMs, yet relatively little work focuses on the security of LLM agents. In this paper, we analyze security and privacy vulnerabilities that are unique to LLM agents. We first provide a taxonomy of attacks categorized by threat actors, objectives, entry points, attacker observability, attack strategies, and inherent vulnerabilities of agent pipelines. We then conduct a series of illustrative attacks on popular open-source and commercial agents, demonstrating the immediate practical implications of their vulnerabilities. Notably, our attacks are trivial to implement and require no understanding of machine learning.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 36 of its 45 references that are in the map. First posted on arXiv 12 Feb 2025.

Builds on

36 of 45 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
  2. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification30 Jul
  3. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies28 Jul
  4. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
  5. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents19 Jun
  6. 2024BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents5 Jun
  7. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents17 Feb
  8. 2023Evil Geniuses: Delving into the Safety of LLM-based Agents20 Nov

Other references · 28

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  4. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  5. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  6. 2022Red Teaming Language Models with Language Modelscited by 983
  7. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  8. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  9. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  10. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  11. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  12. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
16 more
  1. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  2. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  3. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  4. 2020Data Poisoning Attacks Against Federated Learning Systemscited by 266
  5. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  6. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  7. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  8. 2020Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defensescited by 171
  9. 2024The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)cited by 159
  10. 2024Language agents achieve superhuman synthesis of scientific knowledgecited by 121
  11. 2020Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attackscited by 109
  12. 2024Defending LLMs against Jailbreaking Attacks via Backtranslationcited by 65
  13. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  14. 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Modelscited by 42
  15. 2023RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Modelscited by 32
  16. 2023TrojLLM: A Black-box Trojan Prompt Attack on Large Language Modelscited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 29; 2026: 20.

49 papers.

Sort:

By area:

  1. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  2. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 140
  3. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  4. 2025OS-Harm: A Benchmark for Measuring Safety of Computer Use Agentscited by 104
  5. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 93
  6. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 71
  7. 2025Levels of Autonomy for AI Agentscited by 45
  8. 2025RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environmentscited by 42
  9. 2025RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injectioncited by 32
  10. 2025Enforcing Temporal Constraints for LLM Agentscited by 31
  11. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 28
  12. 2025Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysiscited by 27
37 more
  1. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
  2. 2025Building a Foundational Guardrail for General Agentic Systems via Synthetic Datacited by 26
  3. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  4. 2025Real AI Agents with Fake Memories: Fatal Context Manipulation Attacks on Web3 Agentscited by 23
  5. 2025MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environmentscited by 23
  6. 2025A Survey on Agentic Security: Applications, Threats and Defensescited by 17
  7. 2026How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competitioncited by 16
  8. 2026CaMeLs Can Use Computers Too: System-level Security for Computer Use Agentscited by 15
  9. 2025Policy-as-Prompt: Turning AI Governance Rules into Guardrails for AI Agentscited by 12
  10. 2025Invisible Tokens, Visible Bills: The Urgent Need to Audit Hidden Operations in Opaque LLM Servicescited by 10
  11. 2026ACRFence: Preventing Semantic Rollback Attacks in Agent Checkpoint-Restorecited by 9
  12. 2025Takedown: How It's Done in Modern Coding Agent Exploitscited by 6
  13. 2026Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system securitycited by 5
  14. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 5
  15. 2025Unveiling the Landscape of LLM Deployment in the Wild: An Empirical Studycited by 4
  16. 2025Inducing State Anxiety in LLM Agents Reproduces Human-Like Biases in Consumer Decision-Makingcited by 3
  17. 2026Agents at Risk: How Users Unwittingly Undermine LLM Safetycited by 3
  18. 2026ClawTrap: A MITM-Based Red-Teaming Framework for Real-World OpenClaw Security Evaluationcited by 3
  19. 2026Preference Redirection via Attention Concentration: An Attack on Computer Use Agentscited by 3
  20. 2026CIA: Inferring the Communication Topology from LLM-based Multi-Agent Systemscited by 3
  21. 2026Untrusted Content Masking for Web Agents with Security Guaranteescited by 3
  22. 2026Your Agentic LLMs Secretly Encode Indirect Prompt-Injection Exposure in Hidden Statescited by 3
  23. 2025Designing Synthetic Discussion Generation Systems: A Case Study for Online Facilitationcited by 2
  24. 2025SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QAcited by 2
  25. 2025A Red Teaming Roadmap Towards System-Level Safetycited by 2
  26. 2025Throttling Web Agents Using Reasoning Gatescited by 2
  27. 2026Who Pays the Price? Stakeholder-Centric Prompt Injection Benchmarking for Real-world Web Agentscited by 2
  28. 2026LLM-AutoDP: Automatic Data Processing via LLM Agents for Model Fine-tuningcited by 1
  29. 2026AI Security in the Foundation Model Era: A Comprehensive Survey from a Unified Perspectivecited by 1
  30. 2026When the Manual Lies: A Realistic Benchmark to Evaluate MCP Poisoning Attacks for LLM Agentscited by 1
  31. 2026Invisible Ink Threats: Adversarial Goals Behind Legitimate Tasks in Computer-Use Agentscited by 1
  32. 2025Measuring the Security of Mobile LLM Agents under Adversarial Prompts from Untrusted Third-Party Channelscited by 0
  33. 2025CryptoTensors: A Light-Weight Large Language Model File Format for Highly-Secure Model Distributioncited by 0
  34. 2026OS-SPEAR: A Toolkit for the Safety, Performance,Efficiency, and Robustness Analysis of OS Agentscited by 0
  35. 2026Why Trust Your Agent? Empirical Security Gains from TRiSM-Guided Agentic Workflows in Healthcarecited by 0
  36. 2026One Anchor for All: Unified Multilingual and Multimodal Safety Alignment for LVLMscited by 0
  37. 2026The Innocent Courier: Covert Exfiltration Through Legitimate LLM Web Fetchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Models
  2. 2025The Dark Side of LLMs: Agent-based Attack Vectors for System-level Compromise
  3. 2025Unveiling Privacy Risks in LLM Agent Memory
  4. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents
  5. 2025A Survey of Attacks on Large Language Models
  6. 2024Imprompter: Tricking LLM Agents into Improper Tool Use
  7. 2025Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents
  8. 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows