Why LLM Agents Collapse Without Oversight: The Enforcement Gap as the Mechanism Behind Emergence World Failures
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
Binding the audit flag in Reflexion-style agents --- without changing the auditor --- reduces attack success rate substantially, reaching near zero on models whose flags parse cleanly. This single control-flow change exposes the enforcement gap: the controller receives a safety flag and executes anyway. Separating detection probability pd from enforcement probability pe establishes that pe ≈ 0 by default across every framework we tested, making detection quality formally irrelevant to security when enforcement is absent --- a finding consistent with the spontaneous collapses recorded in unsupervised frontier-agent deployments~citepemergence2026. Residual attack success concentrates where flags are unparseable or auditors leak; an RL-trained enforcement controller handles hedged and malformed verdicts that rule-based parsing cannot, cutting ambiguous-critique failure to a fraction of the rule-based baseline. Concurrent filtering and information-flow defenses address detection, not enforcement, leaving the binding constraint untouched. The Audit Enforcement Specification (AES) packages three concrete requirements that close each residue independently; each primitive is adoptable without redesigning the host framework, and no deployed framework currently implements any of them.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 27 of its 38 references that are in the map. First posted on arXiv 14 Sep 2026.
Builds on
27 of 38 references.
Closest ancestors · 5
Same direction, newest first
- 2026ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection27 Aug
- 2026AgentFlow: A Flow-Centric Policy Language and Framework for Securing LLM Agent Systems24 Aug
- 2026SoK: The Attack Surface of Agentic AI - Tools and Autonomy24 Mar
- 2025CommandSans: Securing AI Agents with Surgical Precision Prompt Sanitization9 Oct
- 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep
Other references · 22
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
- 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
10 more
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
- 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
- 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
- 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
- 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
- 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
- 2018AI safety via debatecited by 384
- 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
- 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026When Agents Act Unwatched: The Reduced-Supervision Paradox in Agentic AI
- 2026LLM Agents Can Easily Tamper With Their Own Traces
- 2026Policy-Invisible Violations in LLM-Based Agents
- 2026A GHOST in Long-Horizon Agents: Governance Hazard from Overlooked Safety Constraints across Turns
- 2026Structurally Close, Temporally Distant: Measuring Security Exposure in Long-Horizon LLM Agents
- 2026Isolation as a First-Class Principle for LLM-Agent System Safety: Concepts, Taxonomy, Challenges and Future Directions
- 2025SABER: Small Actions, Big Errors -- Safeguarding Mutating Steps in LLM Agents
- 2025Verifiability-First Agents: Provable Observability and Lightweight Audit Agents for Controlling Autonomous LLM Systems