mlsanityData to 7 Oct 2026

Trust Is Not a Score: Runtime Assurance Contracts for High-Risk AI Agents

arXiv 2609.39717 ↗First posted 30 Sep 2026cs.AI cs.CY cs.SEcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Benchmarks, audits, and agent protocols describe performance, permissions, and repair, but not how observed evidence should change an agent's authority during a consequential task. We call this the assurance-transition gap. We propose a Runtime Assurance Contract (RAC), a policy-level formal schema binding autonomy boundaries, component eligibility, evidence state, transition policy, human-review capacity, and non-compensatory gates. Under RAC, soft metrics may inform routing, whereas a failed or unknown mandatory gate forces retry, switch, escalation, deferral, or stop; aggregate performance cannot authorize action. We define the contract, an evidence record, a permission rule, and five invariants, and illustrate them in clinical, industrial, and judicial failure probes. We then report a deterministic failure-injection study in agentic coding: 280 constructed cases evaluated by a gate conjunction, a score-only rule, and a restricted protocol baseline. At the published example weights and threshold, the score rule admits 80 of 100 block-required injections and all 40 review-required injections. Tuned in hindsight, it matches the conjunction on this corpus. For positive weights, a positive threshold, binary risk signals, zero-signal controls, and an injected case firing each signal alone, we show that exact agreement holds if and only if the threshold does not exceed the smallest weight. A separate set of 18 hand-authored traces checks version-pinned evidence and review transitions against simpler policy variants. In a further prospective synthetic holdout of 24 episodes, two blinded LLM judges assign identical labels to all 72 action attempts; RAC and a separately implemented full stateful baseline both match these labels. These studies test mechanisms on synthetic cases; they establish neither deployed safety nor cross-domain effectiveness.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 21 of its 46 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

21 of 46 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026Do Coding Agents Understand Least-Privilege Authorization?14 May
  2. 2026TRACE: A Metrologically-Grounded Engineering Framework for Trustworthy Agentic AI Systems in Operationally Critical Domains5 May
  3. 2026TraceSafe: A Systematic Assessment of LLM Guardrails on Multi-Step Tool-Calling Trajectories8 Apr
  4. 2026ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedback15 Jan
  5. 2026Towards Verifiably Safe Tool Use for LLM Agents12 Jan
  6. 2025Progent: Securing AI Agents with Privilege Control16 Apr

Other references · 15

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2017Selective Classification for Deep Neural Networkscited by 799
  3. 2017Safe Reinforcement Learning via Shieldingcited by 427
  4. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 333
  5. 2021Guidance on the Assurance of Machine Learning in Autonomous Systems (AMLAS)cited by 40
  6. 2018SOTER: A Runtime Assurance Framework for Programming Safe Robotics Systemscited by 24
  7. 2021Reliability Assessment and Safety Arguments for Machine Learning Components in System Assurancecited by 16
  8. 2017Engineering Trustworthy Self-Adaptive Software with Dynamic Assurance Casescited by 14
  9. 2022Correct-by-Construction Runtime Enforcement in AI -- A Surveycited by 10
  10. 2026Prime Agent: A Self-Improving RLM Harnesscited by 8
  11. 2018Safe Reinforcement Learning via Probabilistic Shieldscited by 7
  12. 2025Tiered Agentic Oversight: A Hierarchical Multi-Agent System for Healthcare Safetycited by 7
3 more
  1. 2022Assurance Cases as Foundation Stone for Auditing AI-enabled and Autonomous Systems: Workshop Results and Political Recommendations for Action from the ExamAI Projectcited by 3
  2. 2026Easy-to-Use Shielding for Reinforcement Learningcited by 1
  3. 2026A Theory of Least Autonomy in AIcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Agent Safety Should Be a Runtime Contract
  2. 2026Agent Behavioral Contracts: Formal Specification and Runtime Enforcement for Reliable Autonomous AI Agents
  3. 2026Engineering Reliable Commit Gates for Agentic AI: Cost-Aware Verification Portfolios under Common-Mode Data Failures
  4. 2026Beyond Task Success: An Evidence-Synthesis Framework for Evaluating, Governing, and Orchestrating Agentic AI
  5. 2026A Trace-Based Assurance Framework for Agentic AI Orchestration: Contracts, Testing, and Governance
  6. 2026Runtime Governance for Agentic AI: Action-Boundary Control with Trusted Provenance and Fail-Closed Execution
  7. 2026Behavioral Governance for Autonomous AI Agents: The AgentBound Framework
  8. 2026Cognitive Admission Control: Risk-Conditioned Assurance for Consequential Actions in Agentic Distributed Systems