mlsanityData to 14 Oct 2026

Position: A Three-Layer Probabilistic Assume-Guarantee Architecture Is Structurally Required for Safe LLM Agent Deployment

arXiv 2605.18672 ↗First posted 18 May 2026cs.AIcited by 1

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

This position paper argues that enforcing LLM agent safety within a single abstraction layer is not merely suboptimal but categorically insufficient for deployed LLM agents -- a structural consequence of how agent execution works, not a contingent limitation of current systems. The three dimensions that jointly constitute safe operation -- semantic intent and policy compliance, environmental validity, and dynamical feasibility -- each depend on a strictly distinct set of information that becomes available at different stages of execution. No single guardrail can certify all three. We argue that the community must respond with a contract-based architecture in which each safety dimension is enforced by an independently certified layer whose probabilistic guarantee satisfies the next layer's assumption. We sketch such an architecture and derive the compositional system-level safety bounds it admits via the chain rule of probability. Three open problems stand between this and a deployable standard: bound estimation from non-i.i.d. traces, graceful degradation of contracts under deployment drift, and extension to multi-agent settings -- the most important unfinished business in LLM agent runtime assurance.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 39 of its 65 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

39 of 65 references.

Closest ancestors · 14

Same direction, newest first

  1. 2026Agent Behavioral Contracts: Formal Specification and Runtime Enforcement for Reliable Autonomous AI Agents25 Feb
  2. 2026Lying with Truths: Open-Channel Multi-Agent Collusion for Belief Manipulation via Generative Montage4 Jan
  3. 2025Enforcing Temporal Constraints for LLM Agents25 Dec
  4. 2025RoboSafe: Safeguarding Embodied Agents via Executable Safety Logic24 Dec
  5. 2025VeriGuard: Enhancing LLM Agent Safety via Verified Code Generation3 Oct
  6. 2025OffTopicEval: When Large Language Models Enter the Wrong Chat, Almost Always!30 Sep
  7. 2025Benchmarking the Robustness of Agentic Systems to Adversarially-Induced Harms22 Aug
  8. 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning26 Mar
  9. 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents24 Mar
  10. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents19 Dec
  11. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents14 Nov
  12. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
2 more
  1. 2024Swiss Cheese Model for AI Safety: A Taxonomy and Reference Architecture for Multi-Layered Guardrails of Foundation Model Based Agents5 Aug
  2. 2023Plug in the Safety Chip: Enforcing Constraints for LLM-driven Robot Agents18 Sep

Other references · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  5. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  6. 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
  7. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  8. 2017Safe Reinforcement Learning via Shieldingcited by 449
  9. 2019Control Barrier Functions: Theory and Applicationscited by 372
  10. 2022Conformal prediction beyond exchangeabilitycited by 355
  11. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 197
  12. 2022Game-theoretic statistics and safe anytime-valid inferencecited by 155
13 more
  1. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 93
  2. 2023Safe Reinforcement Learning via Probabilistic Logic Shieldscited by 42
  3. 2025Safety Guardrails for LLM-Enabled Robotscited by 35
  4. 2024Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Modelscited by 25
  5. 2025Safety Aware Task Planning via Large Language Models in Roboticscited by 24
  6. 2023Robust Conformal Prediction for STL Runtime Verification under Distribution Shiftcited by 15
  7. 2024Your Finetuned Large Language Model is Already a Powerful Out-of-distribution Detectorcited by 11
  8. 2025CluCERT: Certifying LLM Robustness via Clustering-Guided Denoising Smoothingcited by 6
  9. 2025Chain-of-Thought as a Lens: Evaluating Structured Reasoning Alignment between Human Preferences and Large Language Modelscited by 5
  10. 2025Calibrated Predictive Lower Bounds on Time-to-Unsafe-Sampling in LLMscited by 3
  11. 2025Probabilistic Control Barrier Functions: Safety in Probability for Discrete-Time Stochastic Systemscited by 3
  12. 2025ScenicProver: A Framework for Compositional Probabilistic Verification of Learning-Enabled Systemscited by 2
  13. 2025What Does It Take to Get Guarantees? Systematizing Assumptions in Cyber-Physical Systemscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026Toward Safe LLM Agents: A Survey of Specification, Verification, and Enforcement
  2. 2026Safe Multi-Agent Behavior Must Be Maintained, Not Merely Asserted: Constraint Drift in LLM-Based Multi-Agent Systems
  3. 2026Isolation as a First-Class Principle for LLM-Agent System Safety: Concepts, Taxonomy, Challenges and Future Directions
  4. 2025SAFEFLOW: A Principled Protocol for Trustworthy and Transactional Autonomous Agent Systems
  5. 2025Architecting Resilient LLM Agents: A Guide to Secure Plan-then-Execute Implementations
  6. 2025ProbGuard: Proactive Runtime Monitoring for LLM Agent Safety via Probabilistic Prediction
  7. 2026Agent Safety Should Be a Runtime Contract
  8. 2026VEPA: Verified Evolving Formal Policy for Lifelong LLM Agent Guardrails