mlsanityData to 7 Oct 2026

AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents

arXiv 2410.09024 ↗First posted 11 Oct 2024cs.LG cs.AI cs.CLcited by 406

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

The robustness of LLMs to jailbreak attacks, where users design prompts to circumvent safety measures and misuse model capabilities, has been studied primarily for LLMs acting as simple chatbots. Meanwhile, LLM agents -- which use external tools and can execute multi-stage tasks -- may pose a greater risk if misused, but their robustness remains underexplored. To facilitate research on LLM agent misuse, we propose a new benchmark called AgentHarm. The benchmark includes a diverse set of 110 explicitly malicious agent tasks (440 with augmentations), covering 11 harm categories including fraud, cybercrime, and harassment. In addition to measuring whether models refuse harmful agentic requests, scoring well on AgentHarm requires jailbroken agents to maintain their capabilities following an attack to complete a multi-step task. We evaluate a range of leading LLMs, and find (1) leading LLMs are surprisingly compliant with malicious agent requests without jailbreaking, (2) simple universal jailbreak templates can be adapted to effectively jailbreak agents, and (3) these jailbreaks enable coherent and malicious multi-step agent behavior and retain model capabilities. To enable simple and reliable evaluation of attacks and defenses for LLM-based agents, we publicly release AgentHarm at https://huggingface.co/datasets/ai-safety-institute/AgentHarm.

Key facts. Cited by 406 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 33 of its 56 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

33 of 56 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Catastrophic Cyber Capabilities Benchmark (3CB): Robustly Evaluating LLM Agent Cyber Offense Capabilities10 Oct
  2. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification30 Jul
  3. 2024The Dark Side of Function Calling: Pathways to Jailbreaking Large Language Models25 Jul
  4. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
  5. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents19 Jun
  6. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning13 Jun
  7. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  8. 2023Exploiting Novel GPT-4 APIs21 Dec
  9. 2023Evil Geniuses: Delving into the Safety of LLM-based Agents20 Nov
  10. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 23

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  5. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  6. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  7. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  8. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  9. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  10. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  11. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  12. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
11 more
  1. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  2. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 272
  3. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
  4. 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 156
  5. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  6. 2023Emergent autonomous scientific research capabilities of large language modelscited by 127
  7. 2023Evaluating Language-Model Agents on Realistic Autonomous Taskscited by 94
  8. 2023Calibrating LLM-Based Evaluatorcited by 53
  9. 2024PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Railscited by 49
  10. 2024Emerging Vulnerabilities in Frontier Models: Multi-Turn Jailbreak Attackscited by 16
  11. 2024BELLS: A Framework Towards Future Proof Benchmarks for the Evaluation of LLM Safeguardscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 114; 2026: 287.

406 papers.

Sort:

By area:

  1. 2025Humanity's Last Examcited by 467
  2. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  3. 2025Survey on Evaluation of LLM-based Agentscited by 190
  4. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  5. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  6. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  7. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 140
  8. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  9. 2025Agentic Large Language Models, a surveycited by 105
  10. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 105
  11. 2025OS-Harm: A Benchmark for Measuring Safety of Computer Use Agentscited by 104
  12. 2025SafeArena: Evaluating the Safety of Autonomous Web Agentscited by 97
38 more
  1. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 93
  2. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
  3. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1cited by 89
  4. 2025AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agentscited by 79
  5. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 71
  6. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  7. 2025G-Safeguard: A Topology-Guided Security Lens and Treatment on LLM-based Multi-agent Systemscited by 56
  8. 2025Multi-Agent Systems Execute Arbitrary Malicious Codecited by 52
  9. 2024MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Controlcited by 49
  10. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attackscited by 49
  11. 2025RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environmentscited by 42
  12. 2025RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agentscited by 34
  13. 2025Fara-7B: An Efficient Agentic Model for Computer Usecited by 34
  14. 2025Towards Trustworthy GUI Agents: A Surveycited by 33
  15. 2026AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocationscited by 33
  16. 2025Generative to Agentic AI: Survey, Conceptualization, and Challengescited by 32
  17. 2025DoomArena: A framework for Testing AI Agents Against Evolving Security Threatscited by 31
  18. 2025Adversarial Attacks on Robotic Vision Language Action Modelscited by 31
  19. 2025Probing and Steering Evaluation Awareness of Language Modelscited by 31
  20. 2025Enforcing Temporal Constraints for LLM Agentscited by 31
  21. 2026ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspacescited by 31
  22. 2026Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystemscited by 30
  23. 2025The Hawthorne Effect in Reasoning Models: Evaluating and Steering Test Awarenesscited by 29
  24. 2026WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluationcited by 29
  25. 2025LLM Agents Should Employ Security Principlescited by 28
  26. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 28
  27. 2025Adversarial Reasoning at Jailbreaking Timecited by 27
  28. 2025Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysiscited by 27
  29. 2025Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmarkcited by 27
  30. 2025The Jailbreak Tax: How Useful are Your Jailbreak Outputs?cited by 26
  31. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  32. 2025ProbGuard: Proactive Runtime Monitoring for LLM Agent Safety via Probabilistic Predictioncited by 25
  33. 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 24
  34. 2025TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systemscited by 24
  35. 2026SkillAttack: Automated Red Teaming of Agent Skills through Attack Path Refinementcited by 24
  36. 2026ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?cited by 24
  37. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  38. 2025MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environmentscited by 23

50 most cited of 406 All 406 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards a Unified Misuse Monitoring Benchmark
  2. 2026AgentHazard: A Benchmark for Evaluating Harmful Behavior in Computer-Use Agents
  3. 2026Assessing Automated Prompt Injection Attacks in Agentic Environments
  4. 2025Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactions
  5. 2026From Expected Harmfulness to Likelihood: A Probabilistic Reformulation of Jailbreaking LLM Agents
  6. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks
  7. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents
  8. 2026SoK: Rethinking Jailbreaking in the Era of Agentic AI: Attacks, Defenses, and Practical Consideration