Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
Modern LLM agents increasingly rely on context compaction, summarization, or eviction to keep long-running sessions within a token budget. We show that this context-management layer is a safety-critical failure surface: in-context governance constraints that agents reliably obey while visible can be silently removed by compaction, causing the same agent to perform prohibited tool actions later in the session. We call this failure mode Governance Decay. We introduce ConstraintRot, a benchmark of long-horizon agent scenarios with deterministic tool-call grading, and measure compaction-induced violations across seven model families. Across 1,323 episodes, violation rises from 0% with the policy in full context to 30% after compaction, reaching 59% for some models; when the constraint survives the summary, violation remains 0%, but when it is dropped, violation reaches 38%. We further study a Compaction-Eviction Attack, in which adversarial in-context content biases the summarizer to omit a legitimate policy, and show that optimized injections defeat every evaluated model. Finally, we propose Constraint Pinning, a simple training-free mitigation that quarantines governance constraints from lossy compaction and restores violation to 0% in our benchmark. These results identify context management as a first-class governance surface for deployed LLM agents.
Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 31 of its 44 references that are in the map. First posted on arXiv 21 Jun 2026.
Builds on
31 of 44 references.
Closest ancestors · 7
Same direction, newest first
- 2026Omission Constraints Decay While Commission Constraints Persist in Long-Context LLM Agents22 Apr
- 2025When Refusals Fail: Unstable Safety Mechanisms in Long-Context LLM Agents2 Dec
- 2025Defeating Prompt Injections by Design24 Mar
- 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
- 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
- 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training10 Jan
- 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep
Other references · 24
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
12 more
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
- 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
- 2022Red Teaming Language Models with Language Modelscited by 983
- 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 233
- 2025ACON: Optimizing Context Compression for Long-horizon LLM Agentscited by 126
- 2024Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applicationscited by 58
- 2026Beyond Compaction: Structured Context Eviction for Long-Horizon Agentscited by 7
- 2026Slipstream: Trajectory-Grounded Compaction Validation for Long-Horizon Agentscited by 5
- 2026Parallel Context Compaction for Long-Horizon LLM Agent Servingcited by 4
- 2026Constraint Decay: The Fragility of LLM Agents in Backend Code Generationcited by 3
Built on by
Citing papers per year: 2026: 5.
5 papers.
- 2026When Stale Constraints Go Unchecked: Budgeted Verification Failures in Inherited Agent Memorycited by 3
- 2026The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agentscited by 1
- 2026Aborted but Not Forgotten: KV-Cache Retention Breaks Rollback Consistency in Language Agentscited by 0
- 2026Memory as Infrastructure: Reliability Engineering for Persistent Agent Memory in Months-Long LLM-Assisted Developmentcited by 0
- 2026An Architecture for Long-Horizon Agents: Levels, Ticks and Cascaded Intelligencecited by 0
- 2026An Architecture for Long-Horizon Agents: Levels, Ticks and Cascaded Intelligence17 Sep 2026
- 2026Memory as Infrastructure: Reliability Engineering for Persistent Agent Memory in Months-Long LLM-Assisted Development31 Aug 2026
- 2026When Stale Constraints Go Unchecked: Budgeted Verification Failures in Inherited Agent Memory26 Aug 2026
- 2026Aborted but Not Forgotten: KV-Cache Retention Breaks Rollback Consistency in Language Agents16 Aug 2026
- 2026The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agents7 Aug 2026
Similar, not cited
Close in wording, no citation either way
- 2026Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents
- 2026The Compaction Cliff in Long-Running AI Agent Memory
- 2026BLINDSPOT: A Benchmark for Safety and Refusal Calibration in Long-Horizon Tool-Using Agents
- 2026Safe Multi-Agent Behavior Must Be Maintained, Not Merely Asserted: Constraint Drift in LLM-Based Multi-Agent Systems
- 2026A GHOST in Long-Horizon Agents: Governance Hazard from Overlooked Safety Constraints across Turns
- 2026When "Must" Becomes "Maybe": Constraint Weakening in LLM Agent Workflows
- 2025Empirical Characterization of Temporal Constraint Processing in LLMs
- 2026Adversarial Attacks in Multi-Agent LLM Pipelines: Unveiling Structural Vulnerabilities in Agentic AI Architectures