mlsanityData to 14 Oct 2026

Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents

arXiv 2410.13886 ↗First posted 11 Oct 2024cs.CR cs.LGcited by 65

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

For safety reasons, large language models (LLMs) are trained to refuse harmful user instructions, such as assisting dangerous activities. We study an open question in this work: does the desired safety refusal, typically enforced in chat contexts, generalize to non-chat and agentic use cases? Unlike chatbots, LLM agents equipped with general-purpose tools, such as web browsers and mobile devices, can directly influence the real world, making it even more crucial to refuse harmful instructions. In this work, we primarily focus on red-teaming browser agents, LLMs that manipulate information via web browsers. To this end, we introduce Browser Agent Red teaming Toolkit (BrowserART), a comprehensive test suite designed specifically for red-teaming browser agents. BrowserART is consist of 100 diverse browser-related harmful behaviors (including original behaviors and ones sourced from HarmBench [Mazeika et al., 2024] and AirBench 2024 [Zeng et al., 2024b]) across both synthetic and real websites. Our empirical study on state-of-the-art browser agents reveals that, while the backbone LLM refuses harmful instructions as a chatbot, the corresponding agent does not. Moreover, attack methods designed to jailbreak refusal-trained LLMs in the chat settings transfer effectively to browser agents. With human rewrites, GPT-4o and o1-preview-based browser agents attempted 98 and 63 harmful behaviors (out of 100), respectively. We publicly release BrowserART and call on LLM developers, policymakers, and agent developers to collaborate on improving agent safety

Key facts. Cited by 65 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 47 of its 57 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

47 of 57 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage17 Sep
  2. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
  3. 2024Dissecting Adversarial Robustness of Multimodal LM Agents18 Jun
  4. 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilities2 Jun
  5. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents17 Feb
  6. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 41

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  7. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  9. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  11. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  12. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
29 more
  1. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  2. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  3. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  4. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  5. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  6. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  7. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  8. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  9. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  10. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  11. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 363
  12. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 356
  13. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  14. 2023Are aligned neural networks adversarially aligned?cited by 315
  15. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  16. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 223
  17. 2023An Overview of Catastrophic AI Riskscited by 222
  18. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  19. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 176
  20. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 163
  21. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 151
  22. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 144
  23. 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 128
  24. 2024AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policiescited by 74
  25. 2024CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligencecited by 62
  26. 2024HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Human-AI Interactionscited by 44
  27. 2024Leveraging the Context through Multi-Round Interactions for Jailbreaking Attackscited by 36
  28. 2024Evidence of a log scaling law for political persuasion with large language modelscited by 11
  29. 2024A Jailbroken GenAI Model Can Cause Substantial Harm: GenAI-powered Applications are Vulnerable to PromptWarescited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 41; 2026: 21.

65 papers.

Sort:

By area:

  1. 2025Humanity's Last Examcited by 491
  2. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  3. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 157
  4. 2025OS-Harm: A Benchmark for Measuring Safety of Computer Use Agentscited by 111
  5. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 96
  6. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 95
  7. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  8. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 75
  9. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 67
  10. 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languagescited by 59
  11. 2024MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Controlcited by 51
  12. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
38 more
  1. 2025Towards Trustworthy GUI Agents: A Surveycited by 34
  2. 2025DoomArena: A framework for Testing AI Agents Against Evolving Security Threatscited by 32
  3. 2025Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmarkcited by 31
  4. 2026The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systemscited by 30
  5. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 29
  6. 2025Multi-Agent Security Tax: Trading Off Security and Collaboration Capabilities in Multi-Agent Systemscited by 27
  7. 2025Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysiscited by 27
  8. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 24
  9. 2026Your Agent, Their Asset: A Real-World Safety Analysis of OpenClawcited by 20
  10. 2025SecureWebArena: A Holistic Security Evaluation Benchmark for LVLM-based Web Agentscited by 19
  11. 2026AgentWorm: Self-Propagating Attacks Across LLM Agent Ecosystemscited by 17
  12. 2025FocusAgent: Simple Yet Effective Ways of Trimming the Large Context of Web Agentscited by 16
  13. 2025The AI Agent Indexcited by 15
  14. 2025DECEPTICON: How Dark Patterns Manipulate Web Agentscited by 15
  15. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 12
  16. 2025Context manipulation attacks : Web agents are susceptible to corrupted memorycited by 12
  17. 2025Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMscited by 11
  18. 2025AdInject: Real-World Black-Box Attacks on Web Agents via Advertising Deliverycited by 10
  19. 2025Discerning What Matters: A Multi-Dimensional Assessment of Moral Competence in LLMscited by 10
  20. 2025HarmonyGuard: Toward Safety and Utility in Web Agents via Adaptive Policy Enhancement and Dual-Objective Optimizationcited by 10
  21. 2025It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agentscited by 10
  22. 2025Jailbreaking to Jailbreakcited by 9
  23. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 9
  24. 2025Exposing Weak Links in Multi-Agent Systems under Adversarial Promptingcited by 8
  25. 2025Towards Evaluating Proactive Risk Awareness of Multimodal Language Modelscited by 6
  26. 2026Training Agents to Self-Report Misbehaviorcited by 6
  27. 2025Mind the Web: The Security of Web Use Agentscited by 5
  28. 2025Detecting Sleeper Agents in Large Language Models via Semantic Drift Analysiscited by 4
  29. 2025LLMs are Capable of Misaligned Behavior Under Explicit Prohibition and Surveillancecited by 3
  30. 2025Invisible to Humans, Triggered by Agents: Stealthy Jailbreak Attacks on Mobile Vision-Language Agentscited by 3
  31. 2025On the Regulatory Potential of User Interfaces for AI Agent Governancecited by 3
  32. 2026Mind the GAP: Text Safety Does Not Transfer to Tool-Call Safety in LLM Agentscited by 3
  33. 2026Architecture Matters for Multi-Agent Securitycited by 3
  34. 2026Benchmarking Web Agent Safety under E-commerce Deceptive Interfacescited by 3
  35. 2025A Red Teaming Roadmap Towards System-Level Safetycited by 2
  36. 2025Jailbreaking in the Haystackcited by 2
  37. 2026Why Agents Compromise Safety Under Pressurecited by 2
  38. 2026Context-Fractured Decomposition Attacks on Tool-Using LLM Agents: Exploiting Artifact Provenance Gapscited by 2

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM Content Moderation and User Satisfaction: Evidence from Response Refusals in Chatbot Arena
  2. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
  3. 2025Check Yourself Before You Wreck Yourself: Selectively Quitting Improves LLM Agent Safety
  4. 2024Applying Refusal-Vector Ablation to Llama 3.1 70B Agents
  5. 2025When Refusals Fail: Unstable Safety Mechanisms in Long-Context LLM Agents
  6. 2026The Refusal--Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language Models
  7. 2025Echoes of Human Malice in Agents: Benchmarking LLMs for Multi-Turn Online Harassment Attacks
  8. 2025When AI Meets the Web: Prompt Injection Risks in Third-Party AI Chatbot Plugins