# When Refusals Fail: Unstable Safety Mechanisms in Long-Context LLM Agents

arXiv: https://arxiv.org/abs/2512.02445 · HTML: https://mlsanity.com/paper/2512.02445/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 14 of its 17 references that are in the map.
- First posted on arXiv 2 Dec 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 14 of 14)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [Agent-SafetyBench: Evaluating the Safety of LLM Agents](https://mlsanity.com/paper/2412.14470/) (2024)
- [SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal](https://mlsanity.com/paper/2406.14598/) (2024)
- [From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review](https://mlsanity.com/paper/2504.19678/) (2025)
- [Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA](https://mlsanity.com/paper/2406.17419/) (2024)
- [A Comprehensive Survey on Long Context Language Modeling](https://mlsanity.com/paper/2503.17407/) (2025)
- [RedCode: Risky Code Execution and Generation Benchmark for Code Agents](https://mlsanity.com/paper/2411.07781/) (2024)
- [Large Language Model Safety: A Holistic Survey](https://mlsanity.com/paper/2412.17686/) (2024)
- [Position: Agent Should Invoke External Tools ONLY When Epistemically Necessary](https://mlsanity.com/paper/2506.00886/) (2025)

## Built on by (most cited first, 15 of 15)

- [Coding Agents are Effective Long-Context Processors](https://mlsanity.com/paper/2603.20432/) (2026)
- [Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents](https://mlsanity.com/paper/2606.22528/) (2026)
- [Options, Not Clicks: Lattice Refinement for Consent-Driven MCP Authorization](https://mlsanity.com/paper/2605.11360/) (2026)
- [Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents](https://mlsanity.com/paper/2605.16282/) (2026)
- [TRACE: Trajectory Risk-Aware Compression for Long-Horizon Agent Safety](https://mlsanity.com/paper/2606.00611/) (2026)
- [Measuring Harness-Induced Belief Divergence in Multi-Step LLM Agents](https://mlsanity.com/paper/2607.04528/) (2026)
- [Guardrails as Scapegoats: Auditing Unfaithful Safety Refusals in Tool-Augmented LLM Agents](https://mlsanity.com/paper/2607.19449/) (2026)
- [Tool Specifications Matter: Uncovering and Mitigating Safety Risks in AI Agents](https://mlsanity.com/paper/2607.29254/) (2026)
- [State-Dependent Refusal and Learned Incapacity in RLHF-Aligned Language Models](https://mlsanity.com/paper/2512.13762/) (2025)
- [Is Length Really A Liability? An Evaluation of Multi-turn LLM Conversations using BoolQ](https://mlsanity.com/paper/2601.16508/) (2026)
- [AI Guardrail Survival under Single-Cycle Agentic Self-Summarization](https://mlsanity.com/paper/2608.11392/) (2026)
- [AURA-Eval: Evaluation Framework for Acting Under Risk Awareness in LLM Agent Trajectories](https://mlsanity.com/paper/2609.06783/) (2026)
- [Constitutional adapters: Inference-time interventions for misalignment and misuse](https://mlsanity.com/paper/2609.36657/) (2026)
- [Agentic Cognitive Depth: Operational Criteria for Evaluating LLM Agents](https://mlsanity.com/paper/2610.04168/) (2026)
- [Lumen-bench: Cross-Lingual Safety Gaps and the Say-Do Problem in Tool-Calling Large Language Models](https://mlsanity.com/paper/2610.15134/) (2026)

## Abstract (arXiv)

Solving complex or long-horizon problems often requires large language models (LLMs) to use external tools and operate over a significantly longer context window. New LLMs enable longer context windows and support tool calling capabilities. Prior works have focused mainly on evaluation of LLMs on long-context prompts, leaving agentic setup relatively unexplored, both from capability and safety perspectives. Our work addresses this gap. We find that LLM agents could be sensitive to length, type, and placement of the context, exhibiting unexpected and inconsistent shifts in task performance and in refusals to execute harmful requests. Models with 1M-2M token context windows show severe degradation already at 100K tokens, with performance drops exceeding 50% for both benign and harmful tasks. Refusal rates shift unpredictably: GPT-4.1-nano increases from ~5% to ~40% while Grok 4 Fast decreases from ~80% to ~10% at 200K tokens. Our work shows potential safety issues with agents operating on longer context and opens additional questions on the current metrics and paradigm for evaluating LLM agent safety on long multi-step tasks. In particular, our results on LLM agents reveal a notable divergence in both capability and safety performance compared to prior evaluations of LLMs on similar criteria.