# Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agents

arXiv: https://arxiv.org/abs/2602.13379 · HTML: https://mlsanity.com/paper/2602.13379/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 35 of its 68 references that are in the map.
- First posted on arXiv 13 Feb 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 30 of 35)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [Identifying the Risks of LM Agents with an LM-Emulated Sandbox](https://mlsanity.com/paper/2309.15817/) (2023)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [ReTool: Reinforcement Learning for Strategic Tool Use in LLMs](https://mlsanity.com/paper/2504.11536/) (2025)
- [AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents](https://mlsanity.com/paper/2401.13178/) (2024)
- [TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks](https://mlsanity.com/paper/2412.14161/) (2024)
- [An Illusion of Progress? Assessing the Current State of Web Agents](https://mlsanity.com/paper/2504.01382/) (2025)
- [APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay](https://mlsanity.com/paper/2504.03601/) (2025)
- [OpenCUA: Open Foundations for Computer-Use Agents](https://mlsanity.com/paper/2508.09123/) (2025)
- [LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet](https://mlsanity.com/paper/2408.15221/) (2024)
- [ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning](https://mlsanity.com/paper/2503.22738/) (2025)
- [LLM Agents can Autonomously Exploit One-day Vulnerabilities](https://mlsanity.com/paper/2404.08144/) (2024)
- [LlamaFirewall: An open source guardrail system for building secure AI agents](https://mlsanity.com/paper/2505.03574/) (2025)
- [X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents](https://mlsanity.com/paper/2504.13203/) (2025)
- [SafeArena: Evaluating the Safety of Autonomous Web Agents](https://mlsanity.com/paper/2503.04957/) (2025)
- [Teams of LLM Agents can Exploit Zero-Day Vulnerabilities](https://mlsanity.com/paper/2406.01637/) (2024)
- [AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection](https://mlsanity.com/paper/2502.11448/) (2025)
- [The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution](https://mlsanity.com/paper/2510.25726/) (2025)
- [OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety](https://mlsanity.com/paper/2507.06134/) (2025)
- [Speak Out of Turn: Safety Vulnerability of Large Language Models in Multi-turn Dialogue](https://mlsanity.com/paper/2402.17262/) (2024)
- [RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environments](https://mlsanity.com/paper/2505.21936/) (2025)
- [HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Human-AI Interactions](https://mlsanity.com/paper/2409.16427/) (2024)
- [On the Multi-turn Instruction Following for Conversational Web Agents](https://mlsanity.com/paper/2402.15057/) (2024)
- [Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuning](https://mlsanity.com/paper/2410.12952/) (2024)
- [MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use](https://mlsanity.com/paper/2509.24002/) (2025)

## Built on by (most cited first, 30 of 30)

- [Formal Analysis and Supply Chain Security for Agentic AI Skills](https://mlsanity.com/paper/2603.00195/) (2026)
- [TRACES: Proactive Safety Auditing for Multi-Turn LLM Agents via Trajectory-State Modeling](https://mlsanity.com/paper/2605.27690/) (2026)
- [TRACE: Task-Aware Adaptive Self-Evolving Agentic Jailbreaking](https://mlsanity.com/paper/2605.30883/) (2026)
- [Coding with "Enemy": Can Human Developers Detect AI Agent Sabotage?](https://mlsanity.com/paper/2606.05647/) (2026)
- [GuardianAgentBench: Where Agents Fail and How to Guard Them](https://mlsanity.com/paper/2607.20982/) (2026)
- [REDAgentBench: Executable Red Teaming and Faithful Measurement of LLM Agent Systems](https://mlsanity.com/paper/2608.10669/) (2026)
- [STARS: Skill-Triggered Audit for Request-Conditioned Invocation Safety in Agent Systems](https://mlsanity.com/paper/2604.10286/) (2026)
- [Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety](https://mlsanity.com/paper/2605.22643/) (2026)
- [Sell Me This Stock: Unsafe Recommendation Drift in LLM Agents](https://mlsanity.com/paper/2603.12564/) (2026)
- [Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedback](https://mlsanity.com/paper/2605.17453/) (2026)
- [Open-source LLMs administer maximum electric shocks in a Milgram-like obedience experiment](https://mlsanity.com/paper/2605.21401/) (2026)
- [AlphaEval: Evaluating Agents in Production](https://mlsanity.com/paper/2604.12162/) (2026)
- [Silent Failures in Agentic Security Evaluation: A Validated Harness for Tool-Call Mediation Under Indirect Prompt Injection](https://mlsanity.com/paper/2609.32691/) (2026)
- [Representation Transitions Reveal Emerging Safety Risks in Multi-Turn LLM Agents](https://mlsanity.com/paper/2610.00400/) (2026)
- [Signal-Driven Observation for Long-Horizon Web Agents](https://mlsanity.com/paper/2606.06708/) (2026)
- [Hidden in Plain Sight: Benchmarking Agent Safety Against Decomposition Attacks with DECOMPBENCH](https://mlsanity.com/paper/2606.13994/) (2026)
- [RIFT-Bench: Dynamic Red-teaming For Agentic AI Systems](https://mlsanity.com/paper/2606.23927/) (2026)
- [The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems](https://mlsanity.com/paper/2607.19292/) (2026)
- [Hybrid Analysis for Secure MCP Tool Use in LLM Agents](https://mlsanity.com/paper/2607.25297/) (2026)
- [Magnet: Detecting Cross-Session AI Misuse Through Capability Accumulation](https://mlsanity.com/paper/2608.02518/) (2026)
- [Reassembling Distributed Risk: Trajectory-Conditioned Action Generation for Multi-Turn Agent Safety](https://mlsanity.com/paper/2608.25711/) (2026)
- [BLINDSPOT: A Benchmark for Safety and Refusal Calibration in Long-Horizon Tool-Using Agents](https://mlsanity.com/paper/2609.16305/) (2026)
- ["You're Right, Let Me Fix It": How LLM Agents Damage Correct Work When Falsely Accused](https://mlsanity.com/paper/2609.32616/) (2026)
- [Trust the Brand, Lose Control: How Identity Hijacks LLM Agent Orchestration](https://mlsanity.com/paper/2609.32635/) (2026)
- [SEAD: A State-Based Perspective on Attack and Defense in Tool-Using Agents](https://mlsanity.com/paper/2609.34518/) (2026)
- [Jailbreak Context Lingers: Divergent Safety Routing and Its Cross-Task Predictability in Tool Agents](https://mlsanity.com/paper/2609.34686/) (2026)
- [Divide and Inject: Can Agents Reconstruct an Indirect Prompt Injection from Fragments?](https://mlsanity.com/paper/2609.36576/) (2026)
- [PACE: Provenance-Aware Capability Enforcement for Tool-Using LLM Agents](https://mlsanity.com/paper/2610.01349/) (2026)
- [Runtime Authorization of Self-Generated Subgoals in Long-Horizon Tool-Using AI Agents](https://mlsanity.com/paper/2610.04975/) (2026)
- [X-AgentRed: Red-Teaming Personal Agents in Evolving Multi-Service Ecosystems](https://mlsanity.com/paper/2610.13886/) (2026)

## Abstract (arXiv)

LLM-based agents are becoming increasingly capable, yet their safety lags behind. This creates a gap between what agents can do and should do. This gap widens as agents engage in multi-turn interactions and employ diverse tools, introducing new risks overlooked by existing benchmarks. To systematically scale safety testing into multi-turn, tool-realistic settings, we propose a principled taxonomy that transforms single-turn harmful tasks into multi-turn attack sequences. Using this taxonomy, we construct MT-AgentRisk (Multi-Turn Agent Risk Benchmark), the first benchmark to evaluate multi-turn tool-using agent safety. Our experiments reveal substantial safety degradation: the Attack Success Rate (ASR) increases by 16% on average across open and closed models in multi-turn settings. To close this gap, we propose ToolShield, a training-free, tool-agnostic, self-exploration defense: when encountering a new tool, the agent autonomously generates test cases, executes them to observe downstream effects, and distills safety experiences for deployment. Experiments show that ToolShield effectively reduces ASR by 30% on average in multi-turn interactions. Our code is available at https://github.com/CHATS-lab/ToolShield.