mlsanityData to 14 Oct 2026

Agent Safety Should Be a Runtime Contract

arXiv 2608.11274 ↗First posted 11 Aug 2026cs.CR cs.AIcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

The dominant paradigm treats AI safety as a property to be instilled during model training via RLHF, DPO, or Constitutional AI. We argue this is structurally insufficient for autonomous agents that execute code, mutate files, send messages, and modify databases. Agent safety should be a runtime contract enforced by the harness, and the contract has two complementary faces. The preventive face blocks dangerous actions before they happen via sandboxes, permission gates, output filters, and trajectory monitors. The evidential face requires verifiable proof that good actions actually happened, gating task submission on hard evidence such as test runs, log captures, file diffs, and citation grounding. We ground the position in four lines of public evidence, with row-level protocols and data released in the supplementary JSON files: a survey of 52 documented AI-agent and LLM safety incidents, a false-completion audit with 31 non-contested core cases plus one disputed illustrative case, a trajectory-schema audit of 12 public agent systems and harnesses, and a title-level audit of all 28,560 papers accepted at NeurIPS, ICML, and ICLR 2023-2025 showing a pooled 8-12x imbalance between training-time and deployment-time publication. Two prior communities that needed to enforce safety, computer security and the experimental sciences, converged on runtime contracts with both preventive and evidential elements; agentic AI is now under the same pressure. We formalize an Agent Trajectory Schema and Evidence Chain, state a compositional gating proposition based on standard monitor composition, and outline a research agenda. The right unit of safety in agentic AI is the trajectory-with-checkable-evidence, not the model.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 40 of its 92 references that are in the map. First posted on arXiv 11 Aug 2026.

Builds on

40 of 92 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training10 Jan
  2. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 38

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2022Emergent Abilities of Large Language Modelscited by 2,512
  10. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  11. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  12. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
26 more
  1. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  3. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  4. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  5. 2022Toy Models of Superpositioncited by 971
  6. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  7. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  8. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  9. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  10. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  11. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  12. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  13. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
  14. 2024Alignment faking in large language modelscited by 360
  15. 2023Are aligned neural networks adversarially aligned?cited by 315
  16. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  17. 2024Frontier Models are Capable of In-context Schemingcited by 287
  18. 2022Defining and Characterizing Reward Hackingcited by 203
  19. 2026Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnetcited by 191
  20. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 155
  21. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 102
  22. 2020Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Databasecited by 96
  23. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 96
  24. 2025Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Studycited by 49
  25. 2025EchoLeak: The First Real-World Zero-Click Prompt Injection Exploit in a Production LLM Systemcited by 29
  26. 2021D2LV: A Data-Driven and Local-Verification Approach for Image Copy Detectioncited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Trust Is Not a Score: Runtime Assurance Contracts for High-Risk AI Agents
  2. 2026AgentWall: A Runtime Safety Layer for Local AI Agents
  3. 2026AgentTrust: Runtime Safety Evaluation and Interception for AI Agent Tool Use
  4. 2024Safeguarding AI Agents: Developing and Analyzing Safety Architectures
  5. 2026Toward Safe LLM Agents: A Survey of Specification, Verification, and Enforcement
  6. 2025A Safety and Security Framework for Real-World Agentic Systems
  7. 2026AgentS4D: Benchmarking Runtime Risks across the Execution Lifecycle of LLM-Based Workspace Agents
  8. 2026Toward a Principled Framework for Agent Safety Measurement