mlsanityData to 14 Oct 2026

Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents

arXiv 2410.02644 ↗First posted 3 Oct 2024cs.CR cs.AIcited by 369

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Although LLM-based agents, powered by Large Language Models (LLMs), can use external tools and memory mechanisms to solve complex real-world tasks, they may also introduce critical security vulnerabilities. However, the existing literature does not comprehensively evaluate attacks and defenses against LLM-based agents. To address this, we introduce Agent Security Bench (ASB), a comprehensive framework designed to formalize, benchmark, and evaluate the attacks and defenses of LLM-based agents, including 10 scenarios (e.g., e-commerce, autonomous driving, finance), 10 agents targeting the scenarios, over 400 tools, 27 different types of attack/defense methods, and 7 evaluation metrics. Based on ASB, we benchmark 10 prompt injection attacks, a memory poisoning attack, a novel Plan-of-Thought backdoor attack, 4 mixed attacks, and 11 corresponding defenses across 13 LLM backbones. Our benchmark results reveal critical vulnerabilities in different stages of agent operation, including system prompt, user prompt handling, tool usage, and memory retrieval, with the highest average attack success rate of 84.30%, but limited effectiveness shown in current defenses, unveiling important works to be done in terms of agent security for the community. We also introduce a new metric to evaluate the agents' capability to balance utility and security. Our code can be found at https://github.com/agiresearch/ASB.

Key facts. Cited by 369 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 51 of its 78 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

51 of 78 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
  2. 2024BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents5 Jun
  3. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  4. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents17 Feb
  5. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training10 Jan

Other references · 46

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2024Qwen2 Technical Reportcited by 2,120
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  8. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  9. 2024Mixtral of Expertscited by 1,823
  10. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  11. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  12. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
34 more
  1. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  3. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  4. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  5. 2023Detecting Language Model Attacks with Perplexitycited by 431
  6. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 392
  7. 2024Agent Workflow Memorycited by 383
  8. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 364
  9. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  10. 2023Poisoning Language Models During Instruction Tuningcited by 269
  11. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  12. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 258
  13. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  14. 2023OpenAGI: When LLM Meets Domain Expertscited by 197
  15. 2023A Language Agent for Autonomous Drivingcited by 164
  16. 2023FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Designcited by 164
  17. 2023Poisoning Retrieval Corpora by Injecting Adversarial Passagescited by 159
  18. 2024The Impact of Reasoning Step Length on Large Language Modelscited by 153
  19. 2024Certifiably Robust RAG against Retrieval Corruptioncited by 151
  20. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 145
  21. 2020RAB: Provable Robustness Against Backdoor Attackscited by 118
  22. 2024CRAG -- Comprehensive RAG Benchmarkcited by 97
  23. 2023Backdoor Attacks for In-Context Learning with Language Modelscited by 96
  24. 2024BadEdit: Backdooring large language models by model editingcited by 96
  25. 2023Conversational Health Agents: A Personalized LLM-Powered Agent Frameworkcited by 92
  26. 2022Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examplescited by 69
  27. 2022BadPrompt: Backdoor Attacks on Continuous Promptscited by 63
  28. 2023Assessing Prompt Injection Risks in 200+ Custom GPTscited by 55
  29. 2024API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMscited by 39
  30. 2023The Philosopher's Stone: Trojaning Plugins of Large Language Modelscited by 37
  31. 2024PsychoGAT: A Novel Psychological Measurement Paradigm through Interactive Fiction Games with LLM Agentscited by 28
  32. 2023CBD: A Certified Backdoor Detector Based on Local Dominant Probabilitycited by 13
  33. 2024Data-centric NLP Backdoor Defense from the Lens of Memorizationcited by 6
  34. 2022Navigation as Attackers Wish? Towards Building Robust Embodied Agents under Federated Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 73; 2026: 291.

369 papers.

Sort:

By area:

  1. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  2. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 157
  3. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 152
  4. 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoningcited by 151
  5. 2025Progent: Securing AI Agents with Privilege Controlcited by 141
  6. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 133
  7. 2026CoEvoSkills: Self-Evolving Agent Skills via Co-Evolutionary Verificationcited by 128
  8. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 125
  9. 2025Securing AI Agents with Information-Flow Controlcited by 121
  10. 2025OS-Harm: A Benchmark for Measuring Safety of Computer Use Agentscited by 111
  11. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 96
  12. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilitiescited by 96
38 more
  1. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1cited by 91
  2. 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detectioncited by 89
  3. 2025MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agentscited by 77
  4. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 67
  5. 2025A-MemGuard: A Proactive Defense Framework for LLM-Based Agent Memorycited by 67
  6. 2025DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agentscited by 66
  7. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 65
  8. 2025G-Safeguard: A Topology-Guided Security Lens and Treatment on LLM-based Multi-agent Systemscited by 59
  9. 2025ACE: A Security Architecture for LLM-Integrated App Systemscited by 55
  10. 2026Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Frameworkcited by 54
  11. 2025Multi-Agent Systems Execute Arbitrary Malicious Codecited by 53
  12. 2025VeriGuard: Enhancing LLM Agent Safety via Verified Code Generationcited by 49
  13. 2026Memory Poisoning Attack and Defense on Memory Based LLM-Agentscited by 44
  14. 2025MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agentscited by 42
  15. 2026AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocationscited by 38
  16. 2026WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluationcited by 37
  17. 2025RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agentscited by 35
  18. 2025MCIP: Protecting MCP Safety via Model Contextual Integrity Protocolcited by 34
  19. 2025ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agentscited by 31
  20. 2026Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystemscited by 31
  21. 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysiscited by 30
  22. 2026SkillAttack: Automated Red Teaming of Agent Skills through Attack Path Refinementcited by 30
  23. 2025LLM Agents Should Employ Security Principlescited by 29
  24. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 29
  25. 2026From Assistant to Double Agent: Formalizing and Benchmarking Attacks on OpenClaw for Personalized Local AI Agentcited by 29
  26. 2026AgentSentry: Mitigating Indirect Prompt Injection in LLM Agents via Temporal Causal Diagnostics and Context Purificationcited by 29
  27. 2026Taming OpenClaw: Security Analysis and Mitigation of Autonomous LLM Agent Threatscited by 29
  28. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  29. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 28
  30. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 28
  31. 2025Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysiscited by 27
  32. 2025AgentSentinel: An End-to-End and Real-Time Security Defense Framework for Computer-Use Agentscited by 27
  33. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 26
  34. 2025TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systemscited by 26
  35. 2026AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systemscited by 25
  36. 2026SkillTrojan: Backdoor Attacks on Skill-Based Agent Systemscited by 25
  37. 2025BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agentscited by 24
  38. 2026Poison Once, Exploit Forever: Environment-Injected Memory Poisoning Attacks on Web Agentscited by 24

50 most cited of 369 All 369 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks
  2. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks
  3. 2025The Dark Side of LLMs: Agent-based Attack Vectors for System-level Compromise
  4. 2025A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks
  5. 2026On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Models
  6. 2026BackdoorAgent: A Unified Framework for Backdoor Attacks on LLM-based Agents
  7. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
  8. 2026DUMA-Bench: A Dual-Control Multi-Agent Benchmark for Evaluating LLM Agent Security