mlsanityData to 7 Oct 2026

ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedback

arXiv 2601.10156 ↗First posted 15 Jan 2026cs.CLcited by 64

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

While LLM-based agents can interact with environments via invoking external tools, their expanded capabilities also amplify security risks. Monitoring step-level tool invocation behaviors in real time and proactively intervening before unsafe execution is critical for agent deployment, yet remains under-explored. In this work, we first construct TS-Bench, a novel benchmark for step-level tool invocation safety detection in LLM agents. We then develop a guardrail model, TS-Guard, using multi-task reinforcement learning. The model proactively detects unsafe tool invocation actions before execution by reasoning over the interaction history. It assesses request harmfulness and action-attack correlations, producing interpretable and generalizable safety judgments and feedback. Furthermore, we introduce TS-Flow, a guardrail-feedback-driven reasoning framework for LLM agents, which reduces harmful tool invocations of ReAct-style agents by 65 percent on average and improves benign task completion by approximately 10 percent under prompt injection attacks.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 29 of its 39 references that are in the map. First posted on arXiv 15 Jan 2026.

Builds on

29 of 39 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflows28 Oct
  2. 2025Building a Foundational Guardrail for General Agentic Systems via Synthetic Data10 Oct
  3. 2025A-MemGuard: A Proactive Defense Framework for LLM-Based Agent Memory29 Sep
  4. 2025IPIGuard: A Novel Tool Dependency Graph-Based Defense Against Indirect Prompt Injection in LLM Agents21 Aug
  5. 2025AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agents31 May
  6. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks22 Apr
  7. 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning26 Mar
  8. 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection17 Feb
  9. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents19 Dec
  10. 2024AdvAgent: Controllable Blackbox Red-teaming on Web Agents22 Oct
  11. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents19 Jun
  12. 2024BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents5 Jun
1 more
  1. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents17 Feb

Other references · 16

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  3. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  4. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  5. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  6. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  7. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  8. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
  9. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 77
  10. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 58
  11. 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languagescited by 57
  12. 2024InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Modelscited by 36
4 more
  1. 2025LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarkscited by 34
  2. 2025PromptGuard: Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Modelscited by 16
  3. 2024Efficient Universal Goal Hijacking with Semantics-guided Prompt Organizationcited by 13
  4. 2024Quantifying Misalignment Between Agents: Towards a Sociotechnical Understanding of Alignmentcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 64.

64 papers.

Sort:

By area:

  1. 2026SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Taskscited by 45
  2. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 35
  3. 2026AgenTRIM: Tool Risk Mitigation for Agentic AIcited by 9
  4. 2026HINTBench: Horizon-agent Intrinsic Non-attack Trajectory Benchmarkcited by 8
  5. 2026Visual Confused Deputy: Exploiting and Defending Perception Failures in Computer-Using Agentscited by 7
  6. 2026From Governance Norms to Enforceable Controls: A Layered Translation Method for Runtime Guardrails in Agentic AIcited by 7
  7. 2026AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Securitycited by 7
  8. 2026SHE: Trajectory-driven Safety Harness Evolution for LLM Agentscited by 6
  9. 2026StepShield: When, Not Whether to Intervene on Rogue Agentscited by 5
  10. 2026From Risk Classification to Action Plan Remediation: A Guardrail Feedback Driven Framework for LLM Agentscited by 5
  11. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 5
  12. 2026PolicyGuard: A Dialogue-Grounded Sub-Agent Verifier for Policy Adherence in LLM Agentscited by 5
38 more
  1. 2026OpenClawBench: Benchmarking Process-side Anomalies in Real-world Agent Execution Trajectoriescited by 4
  2. 2026MCPShield: A Security Cognition Layer for Adaptive Trust Calibration in Model Context Protocol Agentscited by 3
  3. 2026Oversight Has a Capacity: Calibrating Agent Guards to a Subjective, Fatiguing Humancited by 3
  4. 2026PROJECTMEM: A Local-First, Event-Sourced Memory and Judgment Layer for AI Coding Agentscited by 3
  5. 2026AgentLens: Interpretable Safety Steering via Mechanistic Subspaces for Multi-Turn Coding Agentcited by 3
  6. 2026ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agentscited by 3
  7. 2026DreamGuard: Efficient Runtime Guardrail for LLM Agents via Risk-Aware World Modelcited by 3
  8. 2026LLM-enabled Applications Require System-Level Threat Monitoringcited by 2
  9. 2026Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safetycited by 2
  10. 2026ProMAS: Proactive Error Forecasting for Multi-Agent Systems Using Markov Transition Dynamicscited by 2
  11. 2026Beyond Task Success: An Evidence-Synthesis Framework for Evaluating, Governing, and Orchestrating Agentic AIcited by 2
  12. 2026Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedbackcited by 2
  13. 2026HARP: Measuring Harm Amplification in Multi-Agent LLM Systemscited by 2
  14. 2026RUBAS: Rubric-Based Reinforcement Learning for Agent Safetycited by 2
  15. 2026Outcome Monitors: Recovery Affordances for Silent Tool Failurescited by 2
  16. 2026Plant, Persist, Trigger: Sleeper Attack on Large Language Model Agentscited by 1
  17. 2026SciTrace: Trajectory-Aware Safety Reasoning for Scientific Discovery Agentscited by 1
  18. 2026From Shield to Target: Denial-of-Service Attacks on LLM-Based Agent Guardrailscited by 1
  19. 2026Measuring Harness-Induced Belief Divergence in Multi-Step LLM Agentscited by 1
  20. 2026Isolation as a First-Class Principle for LLM-Agent System Safety: Concepts, Taxonomy, Challenges and Future Directionscited by 1
  21. 2026NEXUS: Structured Runtime Safety for Tool-Using LLM Agentscited by 1
  22. 2026What Can Be Enforced? A Theory of Certified Runtime Safety for Tool-Using Agentscited by 1
  23. 2026LongPIBench: A Long-Context Benchmark for Prompt Injectioncited by 1
  24. 2026trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectoriescited by 1
  25. 2026ScopeBench: Do Agents Preserve Engagement Boundaries Under Goal Pressure?cited by 1
  26. 2026APIOT: Autonomous Vulnerability Management Across Bare-Metal Industrial OT Networkscited by 0
  27. 2026No Action Without a NOD: A Heterogeneous Multi-Agent Architecture for Reliable Service Agentscited by 0
  28. 2026Ghost Tool Calls: Issue-Time Privacy for Speculative Agent Toolscited by 0
  29. 2026Human-on-the-Loop Orchestration for AI-Assisted Legal Discoverycited by 0
  30. 2026SAFETY SENTRY: Context-Aware Human Intervention via EXECUTE-ASK-REFUSE Routingcited by 0
  31. 2026JANUS: Foreseeing Latent Risk for Long-Horizon Agent Safetycited by 0
  32. 2026Where Is the Tradeoff in Using Third-Party API Routers for Agentic Software Development?cited by 0
  33. 2026StealthBench: Measuring Operational Stealth in Autonomous Offensive-Security Agentscited by 0
  34. 2026SkillConsist: Detecting Inconsistencies in Agent Skills via Bidirectional Graph Alignmentcited by 0
  35. 2026Beyond Direct Access: Resource Hijacking in LLM Agentscited by 0
  36. 2026Reassembling Distributed Risk: Trajectory-Conditioned Action Generation for Multi-Turn Agent Safetycited by 0
  37. 2026The Guard That Cried Wolf: How Scary Words Make Agent Guardrails Refuse Legitimate Actionscited by 0
  38. 2026OpenAgentFlow: Enabling System-Wide Safety Boundaries for Heterogeneous AI Agent Fleetscited by 0

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agents
  2. 2026Think Twice Before You Act: Protecting LLM Agents Against Tool Description Poisoning via Isolated Planning
  3. 2026The Verifier Tax: Horizon Dependent Safety Success Tradeoffs in Tool Using LLM Agents
  4. 2025MindGuard: Intrinsic Decision Inspection for Securing LLM Agents Against Metadata Poisoning
  5. 2026StepGuard: Learning Step-Level Guardrails with Scalable Supervision and Safety-Utility Balancing
  6. 2026ChainFuzzer: Greybox Fuzzing for Workflow-Level Multi-Tool Vulnerabilities in LLM Agents
  7. 2026ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis
  8. 2026SafeAgent: A Runtime Protection Architecture for Agentic Systems