mlsanityData to 7 Oct 2026

Identifying the Risks of LM Agents with an LM-Emulated Sandbox

arXiv 2309.15817 ↗First posted 25 Sep 2023cs.AI cs.CL cs.LGcited by 514

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Recent advances in Language Model (LM) agents and tool use, exemplified by applications like ChatGPT Plugins, enable a rich set of capabilities but also amplify potential risks - such as leaking private data or causing financial losses. Identifying these risks is labor-intensive, necessitating implementing the tools, setting up the environment for each test scenario manually, and finding risky cases. As tools and agents become more complex, the high cost of testing these agents will make it increasingly difficult to find high-stakes, long-tailed risks. To address these challenges, we introduce ToolEmu: a framework that uses an LM to emulate tool execution and enables the testing of LM agents against a diverse range of tools and scenarios, without manual instantiation. Alongside the emulator, we develop an LM-based automatic safety evaluator that examines agent failures and quantifies associated risks. We test both the tool emulator and evaluator through human evaluation and find that 68.8% of failures identified with ToolEmu would be valid real-world agent failures. Using our curated initial benchmark consisting of 36 high-stakes tools and 144 test cases, we provide a quantitative risk analysis of current LM agents and identify numerous failures with potentially severe outcomes. Notably, even the safest LM agent exhibits such failures 23.9% of the time according to our evaluator, underscoring the need to develop safer LM agents for real-world deployment.

Key facts. Cited by 514 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 43 of its 86 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

43 of 86 references.

References · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  12. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
31 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  7. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  8. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  9. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  10. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  12. 2022Red Teaming Language Models with Language Modelscited by 983
  13. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  14. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  15. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  16. 2022PAL: Program-aided Language Modelscited by 635
  17. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  18. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  19. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  20. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  21. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  22. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  23. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  24. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 300
  25. 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 277
  26. 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 249
  27. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
  28. 2020LGSVL Simulator: A High Fidelity Simulator for Autonomous Drivingcited by 119
  29. 2019Write, Execute, Assess: Program Synthesis with a REPLcited by 92
  30. 2022Mind's Eye: Grounded Language Model Reasoning through Simulationcited by 83
  31. 2023Collaborating with language models for embodied reasoningcited by 61

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 62; 2025: 103; 2026: 345.

514 papers.

Sort:

By area:

  1. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  2. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  3. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
  4. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  5. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  6. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  7. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  8. 2024Tool Learning with Large Language Models: A Surveycited by 221
  9. 2024AI Agents That Mattercited by 199
  10. 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agentscited by 196
  11. 2025Survey on Evaluation of LLM-based Agentscited by 190
  12. 2023An LLM Compiler for Parallel Function Callingcited by 170
38 more
  1. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 161
  2. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 151
  3. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  4. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  5. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 140
  6. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 129
  7. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  8. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  9. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  10. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 118
  11. 2024Attacking Vision-Language Computer Agents via Pop-upscited by 112
  12. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 112
  13. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  14. 2025OS-Harm: A Benchmark for Measuring Safety of Computer Use Agentscited by 104
  15. 2025PromptArmor: Simple yet Effective Prompt Injection Defensescited by 98
  16. 2025SafeArena: Evaluating the Safety of Autonomous Web Agentscited by 97
  17. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  18. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 93
  19. 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agentscited by 87
  20. 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detectioncited by 82
  21. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 80
  22. 2025AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agentscited by 79
  23. 2024Visibility into AI Agentscited by 77
  24. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 77
  25. 2024Safeguarding Large Language Models: A Surveycited by 73
  26. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  27. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 71
  28. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 70
  29. 2024CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environmentscited by 69
  30. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  31. 2025MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agentscited by 67
  32. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  33. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  34. 2024TrustAgent: Towards Safe and Trustworthy LLM-based Agentscited by 62
  35. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 61
  36. 2026Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agentscited by 57
  37. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 56
  38. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 55

50 most cited of 514 All 514 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026ToolMenuBench: Benchmarking Tool-Menu Filtering Strategies for Reliable and Efficient LLM Agents
  2. 2025More Vulnerable than You Think: On the Stability of Tool-Integrated LLM Agents
  3. 2026ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents
  4. 2025SafetyFlow: An Agent-Flow System for Automated LLM Safety Benchmarking
  5. 2025Tool Preferences in Agentic LLMs are Unreliable
  6. 2026Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety
  7. 2025How Do LLMs Fail In Agentic Scenarios? A Qualitative Analysis of Success and Failure Scenarios of Various LLMs in Agentic Simulations
  8. 2026Risky-Bench: Probing Agentic Safety Risks under Real-World Deployment