The Cold-Start Safety Gap in LLM Agents
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
Are tool-calling LLM agents equally safe throughout a conversation? We discover they are not: agents are most vulnerable at the very start of a session and become substantially safer after a few regular agentic tasks -- a phenomenon we term the cold-start safety gap. To study this systematically, we introduce Safety Over Depth for Agents (SODA), a benchmark that controls how many regular agentic tasks the agent completes before encountering a safety threat, supporting up to 20 preceding tasks. Evaluating 7 models from 4 families, safety improves by 9--52% as the number of preceding regular agentic tasks increases from zero to twenty. Representation analysis confirms that model hidden states gradually shift toward a safety-aligned region as more preceding tasks are present. By systematically studying which part of the preceding conversation matters most, we find that the regular agentic tasks themselves are the primary driver of safety, while the agent's own prior responses have less effect on safety but are essential for preserving later utility. This conclusion is further supported by evaluation on open-source safety benchmarks (AgentHarm, Agent Safety Bench) and utility benchmarks (BFCL, API-Bank), confirming that warming up the agent with regular agentic tasks before deployment makes it safer and preserves full capability. Based on these findings, we recommend a simple deployment strategy: having the agent complete a few regular agentic tasks before possible exposure to safety-critical requests mitigates the cold-start safety gap. Our code is available at https://github.com/Trustworthy-ML-Lab/Agent-Cold-Start-Safety-Gap
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 14 of its 23 references that are in the map. First posted on arXiv 5 Jun 2026.
Builds on
14 of 23 references.
Closest ancestors · 3
Same direction, newest first
- 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents19 Dec
- 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
- 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep
Other references · 11
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
- 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
- 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
- 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
- 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 255
- 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 151
- 2026LLM Agents Already Know When to Call Tools -- Even Without Reasoningcited by 19
- 2024Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilitiescited by 17
- 2025ThinkEdit: Interpretable Weight Editing to Mitigate Overly Short Thinking in Reasoning Modelscited by 17
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents
- 2026Mind the GAP: Text Safety Does Not Transfer to Tool-Call Safety in LLM Agents
- 2025When Refusals Fail: Unstable Safety Mechanisms in Long-Context LLM Agents
- 2026The Causal Impact of Tool Affordance on Safety Alignment in LLM Agents
- 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
- 2025Check Yourself Before You Wreck Yourself: Selectively Quitting Improves LLM Agent Safety
- 2025ManagerBench: Evaluating the Safety-Pragmatism Trade-off in Autonomous LLMs
- 2026Risky-Bench: Probing Agentic Safety Risks under Real-World Deployment