# SafeToolBench: Pioneering a Prospective Benchmark to Evaluating Tool Utilization Safety in LLMs

arXiv: https://arxiv.org/abs/2509.07315 · HTML: https://mlsanity.com/paper/2509.07315/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Sep 2025.

## Built on by (most cited first, 22 of 22)

- [ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis](https://mlsanity.com/paper/2604.02022/) (2026)
- [AgentAbstain: Do LLM Agents Know When Not to Act?](https://mlsanity.com/paper/2607.10059/) (2026)
- [Are Your Agents Upward Deceivers?](https://mlsanity.com/paper/2512.04864/) (2025)
- [FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use](https://mlsanity.com/paper/2603.08262/) (2026)
- [Measuring the Permission Gate: A Stress-Test Evaluation of Claude Code's Auto Mode](https://mlsanity.com/paper/2604.04978/) (2026)
- [AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Security](https://mlsanity.com/paper/2605.29801/) (2026)
- [SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces](https://mlsanity.com/paper/2606.01317/) (2026)
- [LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments](https://mlsanity.com/paper/2605.10779/) (2026)
- [Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents](https://mlsanity.com/paper/2605.16282/) (2026)
- [Memory-Induced Tool-Drift in LLM Agents](https://mlsanity.com/paper/2605.24941/) (2026)
- [Mind the GAP: Text Safety Does Not Transfer to Tool-Call Safety in LLM Agents](https://mlsanity.com/paper/2602.16943/) (2026)
- [Do LLMs Know Tool Irrelevance? Demystifying Structural Alignment Bias in Tool Invocations](https://mlsanity.com/paper/2604.11322/) (2026)
- [Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safety](https://mlsanity.com/paper/2605.12729/) (2026)
- [Hallucination as Exploit: Evidence-Carrying Multimodal Agents](https://mlsanity.com/paper/2605.19192/) (2026)
- [RUBAS: Rubric-Based Reinforcement Learning for Agent Safety](https://mlsanity.com/paper/2606.04051/) (2026)
- [Guardrails as Scapegoats: Auditing Unfaithful Safety Refusals in Tool-Augmented LLM Agents](https://mlsanity.com/paper/2607.19449/) (2026)
- [Failure-Transparent Agents: Benchmarking Post-Failure Reporting in Tool-Using Language Models](https://mlsanity.com/paper/2609.35732/) (2026)
- [ForesightSafety-SAGE:A Fully Automated Scenario Generation and Safety Evaluation Framework for LLM Agents](https://mlsanity.com/paper/2606.08531/) (2026)
- [AURA-Eval: Evaluation Framework for Acting Under Risk Awareness in LLM Agent Trajectories](https://mlsanity.com/paper/2609.06783/) (2026)
- [JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition](https://mlsanity.com/paper/2609.10451/) (2026)
- [PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety](https://mlsanity.com/paper/2609.28197/) (2026)
- [PROACT-Agent: Progressive Runtime Oversight and Active Circuit-breaking for Real-Time Safety](https://mlsanity.com/paper/2609.34415/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have exhibited great performance in autonomously calling various tools in external environments, leading to better problem solving and task automation capabilities. However, these external tools also amplify potential risks such as financial loss or privacy leakage with ambiguous or malicious user instructions. Compared to previous studies, which mainly assess the safety awareness of LLMs after obtaining the tool execution results (i.e., retrospective evaluation), this paper focuses on prospective ways to assess the safety of LLM tool utilization, aiming to avoid irreversible harm caused by directly executing tools. To this end, we propose SafeToolBench, the first benchmark to comprehensively assess tool utilization security in a prospective manner, covering malicious user instructions and diverse practical toolsets. Additionally, we propose a novel framework, SafeInstructTool, which aims to enhance LLMs' awareness of tool utilization security from three perspectives (i.e., textitUser Instruction, Tool Itself, and Joint Instruction-Tool), leading to nine detailed dimensions in total. We experiment with four LLMs using different methods, revealing that existing approaches fail to capture all risks in tool utilization. In contrast, our framework significantly enhances LLMs' self-awareness, enabling a more safe and trustworthy tool utilization.