mlsanityData to 14 Oct 2026

SafeArena: Evaluating the Safety of Autonomous Web Agents

arXiv 2503.04957 ↗First posted 6 Mar 2025cs.LG cs.AI cs.CLcited by 102

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

LLM-based agents are becoming increasingly proficient at solving web-based tasks. With this capability comes a greater risk of misuse for malicious purposes, such as posting misinformation in an online forum or selling illicit substances on a website. To evaluate these risks, we propose SafeArena, the first benchmark to focus on the deliberate misuse of web agents. SafeArena comprises 250 safe and 250 harmful tasks across four websites. We classify the harmful tasks into five harm categories -- misinformation, illegal activity, harassment, cybercrime, and social bias, designed to assess realistic misuses of web agents. We evaluate leading LLM-based web agents, including GPT-4o, Claude-3.5 Sonnet, Qwen-2-VL 72B, and Llama-3.2 90B, on our benchmark. To systematically assess their susceptibility to harmful tasks, we introduce the Agent Risk Assessment framework that categorizes agent behavior across four risk levels. We find agents are surprisingly compliant with malicious requests, with GPT-4o and Qwen-2 completing 34.7% and 27.3% of harmful requests, respectively. Our findings highlight the urgent need for safety alignment procedures for web agents. Our benchmark is available here: https://safearena.github.io

Key facts. Cited by 102 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 32 of its 50 references that are in the map. First posted on arXiv 6 Mar 2025.

Builds on

32 of 50 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
  2. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents9 Oct
  3. 2024The Dark Side of Function Calling: Pathways to Jailbreaking Large Language Models25 Jul
  4. 2024Dissecting Adversarial Robustness of Multimodal LM Agents18 Jun
  5. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  6. 2024WIPI: A New Web Threat for LLM-Driven Web Agents26 Feb
  7. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  7. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  8. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  9. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  10. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  11. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  12. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
13 more
  1. 2023Language Models can Solve Computer Taskscited by 372
  2. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 363
  3. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  4. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  5. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 284
  6. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 180
  7. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 161
  8. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 160
  9. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 121
  10. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 68
  11. 2023SteP: Stacked LLM Policies for Web Actionscited by 66
  12. 2024Multi-Turn Context Jailbreak Attack on Large Language Models From First Principlescited by 56
  13. 2023Bypassing the Safety Training of Open-Source LLMs with Priming Attackscited by 50

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 38; 2026: 64.

102 papers.

Sort:

By area:

  1. 2025Survey on Evaluation of LLM-based Agentscited by 204
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  3. 2025OS-Harm: A Benchmark for Measuring Safety of Computer Use Agentscited by 111
  4. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 75
  5. 2025Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agentscited by 62
  6. 2025CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactionscited by 47
  7. 2025RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environmentscited by 46
  8. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 45
  9. 2025RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agentscited by 35
  10. 2025DoomArena: A framework for Testing AI Agents Against Evolving Security Threatscited by 32
  11. 2026Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agentscited by 30
  12. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 28
38 more
  1. 2025TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systemscited by 26
  2. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  3. 2025WebGuard: Building a Generalizable Guardrail for Web Agentscited by 21
  4. 2025A Survey on Agentic Security: Applications, Threats and Defensescited by 20
  5. 2025SecureWebArena: A Holistic Security Evaluation Benchmark for LVLM-based Web Agentscited by 19
  6. 2025Build the web for agents, not agents for the webcited by 18
  7. 2025FocusAgent: Simple Yet Effective Ways of Trimming the Large Context of Web Agentscited by 16
  8. 2025Are Your Agents Upward Deceivers?cited by 16
  9. 2025WAREX: Web Agent Reliability Evaluation on Existing Benchmarkscited by 15
  10. 2026The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agentscited by 12
  11. 2026FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Usecited by 10
  12. 2025Just Do It!? Computer-Use Agents Exhibit Blind Goal-Directednesscited by 9
  13. 2026LPS-Bench: Benchmarking Safety Awareness of Computer-Use Agents in Long-Horizon Planning under Benign and Adversarial Scenarioscited by 9
  14. 2026HINTBench: Horizon-agent Intrinsic Non-attack Trajectory Benchmarkcited by 9
  15. 2025Build Agent Advocates, Not Platform Agentscited by 8
  16. 2025Multimodal Cultural Safety: Evaluation Framework and Alignment Strategiescited by 8
  17. 2025STAC: When Innocent Tools Form Dangerous Chains for LLM Agentscited by 8
  18. 2025Exposing Weak Links in Multi-Agent Systems under Adversarial Promptingcited by 8
  19. 2026Risky-Bench: Probing Agentic Safety Risks under Real-World Deploymentcited by 8
  20. 2026TRACE: Trajectory-Aware Comprehensive Evaluation for Deep Research Agentscited by 8
  21. 2026AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Securitycited by 8
  22. 2025Graph2Eval: Automatic Multimodal Task Generation for Agents via Knowledge Graphscited by 7
  23. 2026StepShield: When, Not Whether to Intervene on Rogue Agentscited by 7
  24. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 7
  25. 2026Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verificationcited by 7
  26. 2025Towards Evaluating Proactive Risk Awareness of Multimodal Language Modelscited by 6
  27. 2025Environmental Injection Attacks against GUI Agents in Realistic Dynamic Environmentscited by 6
  28. 2026PhoneHarness: Harnessing Phone-Use Agents through Mixed GUI, CLI, and Tool Actionscited by 6
  29. 2025Mobile GUI Agents under Real-world Threats: Are We There Yet?cited by 5
  30. 2025Your Harness is Not Secure: Benchmarking Real-world Threat of Command Line Interface Agentcited by 5
  31. 2025SIRAJ: Diverse and Efficient Red-Teaming for LLM Agents via Distilled Structured Reasoningcited by 5
  32. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
  33. 2026Understanding and Evaluating Claw-like Agent Security Through a Computer-Systems Lenscited by 5
  34. 2026Spinning Straw into Gold: Relabeling LLM Agent Trajectories in Hindsight for Successful Demonstrationscited by 5
  35. 2026Can LLM Agents Be CFOs? Benchmarking Long-Horizon Resource Allocation in an Uncertain Enterprise Environmentcited by 4
  36. 2026The Authorization-Execution Gap Is a Major Safety and Security Problem in Open-World Agentscited by 4
  37. 2026AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptionscited by 4
  38. 2026PhoneBuddy: Training Open Models for Agentic Phone Usecited by 4

50 most cited of 102 All 102 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026HarmfulSkillBench: How Do Harmful Skills Weaponize Your Agents?
  2. 2026WAAA! Web Adversaries Against Agentic Browsers
  3. 2026AgentHazard: A Benchmark for Evaluating Harmful Behavior in Computer-Use Agents
  4. 2023Testing Language Model Agents Safely in the Wild
  5. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities
  6. 2025SafeAgent: Safeguarding LLM Agents via an Automated Risk Simulator
  7. 2026OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills
  8. 2026MalURLBench: A Benchmark Evaluating Agents' Vulnerabilities When Processing Web URLs