mlsanityData to 14 Oct 2026

AgentGuard: Runtime Verification of AI Agents

arXiv 2509.23864 ↗First posted 28 Sep 2025cs.AI cs.SEcited by 17

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

The rapid evolution to autonomous, agentic AI systems introduces significant risks due to their inherent unpredictability and emergent behaviors; this also renders traditional verification methods inadequate and necessitates a shift towards probabilistic guarantees where the question is no longer if a system will fail, but the probability of its failure within given constraints. This paper presents AgentGuard, a framework for runtime verification of Agentic AI systems that provides continuous, quantitative assurance through a new paradigm called Dynamic Probabilistic Assurance. AgentGuard operates as an inspection layer that observes an agent's raw I/O and abstracts it into formal events corresponding to transitions in a state model. It then uses online learning to dynamically build and update a Markov Decision Process (MDP) that formally models the agent's emergent behavior. Using probabilistic model checking, the framework then verifies quantitative properties in real-time.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 19 of its 28 references that are in the map. First posted on arXiv 28 Sep 2025.

Builds on

19 of 28 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systems4 Jun

Other references · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024RepairAgent: An Autonomous, LLM-Based Agent for Program Repaircited by 232
  3. 2024Formal-LLM: Integrating Formal Language and Natural Language for Controllable LLM-based Agentscited by 40
  4. 2021A Review of Formal Methods applied to Machine Learningcited by 33
  5. 2024A Survey on Context-Aware Multi-Agent Systems: Techniques, Challenges and Future Directionscited by 27
  6. 2024The Fusion of Large Language Models and Formal Methods for Trustworthy AI Agents: A Roadmapcited by 20
  7. 2025VeriPlan: Integrating Formal Verification and LLMs into End-User Planningcited by 20
  8. 2025Agentic AI Needs a Systems Theorycited by 19
  9. 2025AI Agents: Evolution, Architecture, and Real-World Applicationscited by 19
  10. 2025Safe: Enhancing Mathematical Reasoning in Large Language Models via Retrospective Step-aware Formal Verificationcited by 17
  11. 2023Formally Specifying the High-Level Behavior of LLM-Based Agentscited by 15
  12. 2021Probabilistic Model Checking and Autonomycited by 9
6 more
  1. 2025Saarthi: The First AI Formal Verification Engineercited by 8
  2. 2023Fully Automatic Neural Network Reduction for Formal Verificationcited by 6
  3. 2025Supporting Software Formal Verification with Large Language Models: An Experimental Studycited by 6
  4. 2025Step-Wise Formal Verification for LLM-Based Mathematical Problem Solvingcited by 5
  5. 2025Hey AI, Generate Me a Hardware Code! Agentic AI-based Hardware Design & Verificationcited by 5
  6. 2025A Short Survey on Formalising Software Requirements using Large Language Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 16.

17 papers.

Sort:

By area:

  1. 2026AgentRx: Diagnosing AI Agent Failures from Execution Trajectoriescited by 60
  2. 2026SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Taskscited by 55
  3. 2026From Fluent to Verifiable: Claim-Level Auditability for Deep Research Agentscited by 7
  4. 2025Agentic Services Computingcited by 6
  5. 2026Traversal-as-Policy: Log-Distilled Gated Behavior Trees as Externalized, Verifiable Policies for Safe, Robust, and Efficient Agentscited by 5
  6. 2026TriCEGAR: A Trace-Driven Abstraction Mechanism for Agentic AIcited by 2
  7. 2026Toward Safe LLM Agents: A Survey of Specification, Verification, and Enforcementcited by 2
  8. 2026From Thinker to Society: Security in Hierarchical Autonomy Evolution of AI Agentscited by 1
  9. 2026Alignment Contracts for Agentic Security Systemscited by 1
  10. 2026AIVV: Neuro-Symbolic LLM Agent-Integrated Verification and Validation for Trustworthy Autonomous Systemscited by 0
  11. 2026From CRUD to Autonomous Agents: Formal Validation and Zero-Trust Security for Semantic Gateways in AI-Native Enterprise Systemscited by 0
  12. 2026TraceFix: Repairing Agent Coordination Protocols with TLA+ Counterexamplescited by 0
5 more
  1. 2026Ceci n'est pas une pipe: AI systems as semantic abstractionscited by 0
  2. 2026Beyond Component Testing: Validating Agentic AI Systemscited by 0
  3. 2026ATLAS: Discovering Agent Strategies through LLM-Guided Abstraction and Automata Learningcited by 0
  4. 2026Agents as Software: A Programming Languages Agenda for Agent Reliabilitycited by 0
  5. 2026Sentry: Learning to Recover from LLM Agent Failures at Test Timecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ProbGuard: Proactive Runtime Monitoring for LLM Agent Safety via Probabilistic Prediction
  2. 2026Efficient and Sound Probabilistic Verification for AI Agents
  3. 2026One Monitor Does Not Fit All: Formalism-Agnostic Runtime Verification for Agentic AI
  4. 2026Engineering Reliable Commit Gates for Agentic AI: Cost-Aware Verification Portfolios under Common-Mode Data Failures
  5. 2025Are Agents Probabilistic Automata? A Trace-Based, Memory-Constrained Theory of Agentic AI
  6. 2026Trust Is Not a Score: Runtime Assurance Contracts for High-Risk AI Agents
  7. 2026Agent Behavioral Contracts: Formal Specification and Runtime Enforcement for Reliable Autonomous AI Agents
  8. 2026AgentWard: A Lifecycle Security Architecture for Autonomous AI Agents