mlsanityData to 14 Oct 2026

Testing Language Model Agents Safely in the Wild

arXiv 2311.10538 ↗First posted 17 Nov 2023cs.AIcited by 46

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

A prerequisite for safe autonomy-in-the-wild is safe testing-in-the-wild. Yet real-world autonomous tests face several unique safety challenges, both due to the possibility of causing harm during a test, as well as the risk of encountering new unsafe agent behavior through interactions with real-world and potentially malicious actors. We propose a framework for conducting safe autonomous agent tests on the open internet: agent actions are audited by a context-sensitive monitor that enforces a stringent safety boundary to stop an unsafe test, with suspect behavior ranked and logged to be examined by humans. We design a basic safety monitor (AgentMonitor) that is flexible enough to monitor existing LLM agents, and, using an adversarial simulated agent, we measure its ability to identify and stop unsafe situations. Then we apply the AgentMonitor on a battery of real-world tests of AutoGPT, and we identify several limitations and challenges that will face the creation of safe in-the-wild tests as autonomous agents grow more capable.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 12 of its 25 references that are in the map. First posted on arXiv 17 Nov 2023.

Builds on

12 of 25 references.

References · 12

  1. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  3. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  4. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  5. 2023Large Language Modelscited by 780
  6. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  7. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  8. 2023Do Large Language Models Know What They Don't Know?cited by 243
  9. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  10. 2023An Overview of Catastrophic AI Riskscited by 222
  11. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 182
  12. 2020Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real Worldcited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 14; 2025: 16; 2026: 15.

46 papers.

Sort:

By area:

  1. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
  2. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 229
  3. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 171
  4. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
  5. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  6. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
  7. 2025PromptArmor: Simple yet Effective Prompt Injection Defensescited by 108
  8. 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agentscited by 90
  9. 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detectioncited by 89
  10. 2024Detoxifying Large Language Models via Knowledge Editingcited by 87
  11. 2026SkillClaw: Let Skills Evolve Collectively with Agentic Evolvercited by 81
  12. 2024Visibility into AI Agentscited by 79
34 more
  1. 2024Safeguarding Large Language Models: A Surveycited by 78
  2. 2025MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agentscited by 77
  3. 2024TrustAgent: Towards Safe and Trustworthy LLM-based Agentscited by 64
  4. 2024Large Language Model Safety: A Holistic Surveycited by 59
  5. 2024Risks of AI Scientists: Prioritizing Safeguarding Over Autonomycited by 49
  6. 2026Building Production-Ready Probes For Geminicited by 47
  7. 2026MetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the Wildcited by 37
  8. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 28
  9. 2025Monitoring Decomposition Attacks in LLMs with Lightweight Sequential Monitorscited by 25
  10. 2024Testing and Understanding Erroneous Planning in LLM Agents through Synthesized User Inputscited by 22
  11. 2025WebGuard: Building a Generalizable Guardrail for Web Agentscited by 21
  12. 2025OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflowscited by 19
  13. 2025ToolFuzz -- Automated Agent Tool Testingcited by 18
  14. 2025Think Twice Before You Act: Enhancing Agent Behavioral Safety with Thought Correctioncited by 14
  15. 2026FORTIS: Benchmarking Over-Privilege in Agent Skillscited by 13
  16. 2026AgentGuardian: Learning Access Control Policies to Govern AI Agent Behaviorcited by 11
  17. 2025Mimicking the Familiar: Dynamic Command Generation for Information Theft Attacks in LLM Tool-Learning Systemcited by 10
  18. 2025RedCodeAgent: Automatic Red-teaming Agent against Diverse Code Agentscited by 9
  19. 2024Autonomous Workflow for Multimodal Fine-Grained Training Assistants Towards Mixed Realitycited by 8
  20. 2025Resource Rational Contractualism Should Guide AI Alignmentcited by 8
  21. 2024The Role of Governments in Increasing Interconnected Post-Deployment Monitoring of AIcited by 7
  22. 2025SafeAgent: Safeguarding LLM Agents via an Automated Risk Simulatorcited by 6
  23. 2026Training Agents to Self-Report Misbehaviorcited by 6
  24. 2026Capability Minimization as a Safety Primitive: Risk-Aware Causal Gating for Least-Privilege LLM Agentscited by 6
  25. 2025LumiMAS: A Comprehensive Framework for Real-Time Monitoring and Enhanced Observability in Multi-Agent Systemscited by 4
  26. 2026BeSafe-Bench: Unveiling Behavioral Safety Risks of Situated Agents in Functional Environmentscited by 4
  27. 2026Simulating Complex Multi-Turn Tool Calling Interactions in Stateless Execution Environmentscited by 3
  28. 2026SafeMCP: Proactive Power Regulation for LLM Agent Defense via Environment-Grounded Look-Ahead Reasoningcited by 3
  29. 2025AgentCrypt: Advancing Privacy and (Secure) Computation in AI Agent Collaborationcited by 2
  30. 2025Risks of AI-driven product development and strategies for their mitigationcited by 1
  31. 2026Execution-Grounded Security Testing for Coding Agents in Software Engineering Pipelinescited by 0
  32. 2026Safety Does Not Compose: Non-Decaying Loop State for Autonomous LLM Agentscited by 0
  33. 2026AURA-Eval: Evaluation Framework for Acting Under Risk Awareness in LLM Agent Trajectoriescited by 0
  34. 2026Universal Defenses for Tool-Integrated LLM Agents Against Adversarial Attackscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety
  2. 2026AgentWall: A Runtime Safety Layer for Local AI Agents
  3. 2026Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safety
  4. 2026ClawSentry: A Progressive Multi-Tier Security Monitor for Safeguarding Autonomous LLM Agents
  5. 2024Safeguarding AI Agents: Developing and Analyzing Safety Architectures
  6. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents
  7. 2026Toward a Principled Framework for Agent Safety Measurement
  8. 2025SafeArena: Evaluating the Safety of Autonomous Web Agents