Operational Hallucination and Safety Drift in AI Agents
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
Large language models (LLMs) serving as planners in tool-using autonomous agents introduce dynamic reliability risks in multi-turn execution. While single-turn safety mechanisms are relatively mature, extended interactions reveal structural vulnerabilities where initial alignment degrades over time. This paper empirically characterizes two observed failure modes across multiple state-of-the-art LLMs: Safety Drift, the gradual erosion of declared safety intent leading to constraint-violating actions (e.g., textual refusal followed by reconnaissance and unsafe execution), and Operational Hallucination, persistent repetitive tool calls indicative of flawed state perception (e.g., livelocks even in legitimate tasks). Through controlled multi-turn evaluation on high-stakes ethical dilemmas, malicious requests, and benign controls, we quantify these phenomena using declaration-action gap and livelock metrics, demonstrating their cross-model prevalence under direct execution protocols. Root-cause analysis attributes the instabilities to the decoupling of reasoning context from execution state in current agent loops. We propose an Action-Aware Supervision Layer - a lightweight, plug-and-play architectural blueprint incorporating intent-action consistency checks, runtime state tracking, and forced termination primitives. Post-hoc simulation on captured failure trajectories shows the layer can intercept observed violations without false positives on benign cases. This work advances agent reliability by shifting focus from linguistic safeguards to enforceable architectural mechanisms for responsible agentic AI.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 14 of its 24 references that are in the map. First posted on arXiv 20 Jul 2026.
Builds on
14 of 24 references.
Closest ancestors · 1
Same direction, newest first
Other references · 13
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
- 2016Concrete Problems in AI Safetycited by 1,582
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
- 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
- 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 128
- 2023Eight Things to Know about Large Language Modelscited by 73
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Tool Specifications Matter: Uncovering and Mitigating Safety Risks in AI Agents
- 2026Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety
- 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
- 2026AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents
- 2026A GHOST in Long-Horizon Agents: Governance Hazard from Overlooked Safety Constraints across Turns
- 2026PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety
- 2025When Refusals Fail: Unstable Safety Mechanisms in Long-Context LLM Agents
- 2026BLINDSPOT: A Benchmark for Safety and Refusal Calibration in Long-Horizon Tool-Using Agents