What Can Be Enforced? A Theory of Certified Runtime Safety for Tool-Using Agents
On the map: LLM safety and security → Prompt injection and agent security
Abstract (arXiv)
Runtime guardrails act before irreversible tool calls, but their guarantees depend on what policy state is representable, what a judge observes, and whether intervention changes future behavior. We separate three questions. First, relative to fixed oracle predicates, a deterministic gate enforces exactly the nonempty safety policies whose good prefixes its register model recognizes; policy nontriviality is undecidable with two decrementable counters but in PSPACE for a separable monotone fragment. Second, under a fixed exogenous law, Neyman-Pearson gives the exact false-block/miss frontier and conformal calibration gives a finite-sample marginal certificate, possibly via block-all. Third, once blocking changes future proposals, static scores and ungated trajectories need not identify the closed-loop frontier; a specified finite controlled model instead yields an occupancy program. Bounded representation attacks add a robustness margin, so benign calibration alone does not transfer. Experiments target these distinctions through static diagnostics, controlled-model enumeration, representation rewrites, and paired closed-loop reruns.
Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 21 of its 60 references that are in the map. First posted on arXiv 24 Jul 2026.
Builds on
21 of 60 references.
Closest ancestors · 9
Same direction, newest first
- 2026From Risk Classification to Action Plan Remediation: A Guardrail Feedback Driven Framework for LLM Agents4 Jun
- 2026ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedback15 Jan
- 2025Enforcing Temporal Constraints for LLM Agents25 Dec
- 2025Adversarial Attacks on LLM-as-a-Judge Systems: Insights from Prompt Injections25 Apr
- 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning26 Mar
- 2025Defeating Prompt Injections by Design24 Mar
- 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection17 Feb
- 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
- 2024Games for AI Control: Models of Safety Evaluations of AI Deployment Protocols12 Sep
Other references · 12
- 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
- 2017Selective Classification for Deep Neural Networkscited by 852
- 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
- 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 308
- 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 181
- 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 96
- 2023LLM Censorship: A Machine Learning Challenge or a Computer Security Problem?cited by 49
- 2025VeriGuard: Enhancing LLM Agent Safety via Verified Code Generationcited by 49
- 2015Shield Synthesis: Runtime Enforcement for Reactive Systemscited by 42
- 2016Neyman-Pearson (NP) classification algorithms and NP receiver operating characteristics (NP-ROC)cited by 23
- 2022Conformal Quantitative Predictive Monitoring of STL Requirements for Stochastic Processescited by 13
- 2026Oversight Has a Capacity: Calibrating Agent Guards to a Subjective, Fatiguing Humancited by 4
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026Stale Does Not Mean Unsafe: Guard Precision for Tool-Using LLM Agents under Infrastructure State Racescited by 0
- 2026Evaluate the Stack, Not the Layer: Do Deterministic and LLM Gates for Agent Actions Fail Independently?cited by 0
- 2026SpecGuard: Proving a Task Is Broken Before the Agent Cheatscited by 0
- 2026agent-risk: Risk Assessment for Changes to AI Agentscited by 0
- 2026agent-risk: Risk Assessment for Changes to AI Agents12 Oct 2026
- 2026SpecGuard: Proving a Task Is Broken Before the Agent Cheats6 Oct 2026
- 2026Evaluate the Stack, Not the Layer: Do Deterministic and LLM Gates for Agent Actions Fail Independently?5 Oct 2026
- 2026Stale Does Not Mean Unsafe: Guard Precision for Tool-Using LLM Agents under Infrastructure State Races25 Aug 2026
Similar, not cited
Close in wording, no citation either way
- 2026Guardrails as Infrastructure: Policy-First Control for Tool-Orchestrated Workflows
- 2026The Verifier Tax: Horizon Dependent Safety Success Tradeoffs in Tool Using LLM Agents
- 2026Agent Safety Should Be a Runtime Contract
- 2026VIGIL: Runtime Enforcement of Behavioral Specifications in AI Agent Skills
- 2026Environment Steering: Using Data Flow Control to Improve Agent Utility and Safety
- 2025QuadSentinel: Sequent Safety for Machine-Checkable Control in Multi-agent Systems
- 2026CAGE: Certified Authorization under Typed-Return Uncertainty for Tool-Using Agents
- 2026Falsifiable Release Gates for Self-Improving Systems: Standing Invariants at Scale