# ProbGuard: Proactive Runtime Monitoring for LLM Agent Safety via Probabilistic Prediction

arXiv: https://arxiv.org/abs/2508.00500 · HTML: https://mlsanity.com/paper/2508.00500/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 19 of its 53 references that are in the map.
- First posted on arXiv 1 Aug 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 19 of 19)

- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents](https://mlsanity.com/paper/2410.09024/) (2024)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents](https://mlsanity.com/paper/2503.18666/) (2025)
- [Survey on Evaluation of LLM-based Agents](https://mlsanity.com/paper/2503.16416/) (2025)
- [ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning](https://mlsanity.com/paper/2503.22738/) (2025)
- [GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning](https://mlsanity.com/paper/2406.09187/) (2024)
- [SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents](https://mlsanity.com/paper/2412.13178/) (2024)
- [A Review of Prominent Paradigms for LLM-Based Agents: Tool Use (Including RAG), Planning, and Feedback Learning](https://mlsanity.com/paper/2406.05804/) (2024)
- [Conformal Prediction for STL Runtime Verification](https://mlsanity.com/paper/2211.01539/) (2022)
- [LLM-Empowered Embodied Agent for Memory-Augmented Task Planning in Household Robotics](https://mlsanity.com/paper/2504.21716/) (2025)
- [LawBreaker: An Approach for Specifying Traffic Laws and Fuzzing Autonomous Vehicles](https://mlsanity.com/paper/2208.14656/) (2022)
- [REDriver: Runtime Enforcement for Autonomous Vehicles](https://mlsanity.com/paper/2401.02253/) (2024)
- [Learning-Based Approaches to Predictive Monitoring with Conformal Statistical Guarantees](https://mlsanity.com/paper/2312.01959/) (2023)
- [PSTMonitor: Monitor Synthesis from Probabilistic Session Types](https://mlsanity.com/paper/2212.07329/) (2022)

## Built on by (most cited first, 28 of 28)

- [ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis](https://mlsanity.com/paper/2604.02022/) (2026)
- [Agent Behavioral Contracts: Formal Specification and Runtime Enforcement for Reliable Autonomous AI Agents](https://mlsanity.com/paper/2602.22302/) (2026)
- [A Framework for Formalizing LLM Agent Security](https://mlsanity.com/paper/2603.19469/) (2026)
- [RoboSafe: Safeguarding Embodied Agents via Executable Safety Logic](https://mlsanity.com/paper/2512.21220/) (2025)
- [ToolPrivacyBench: Benchmarking Purpose-Bound Privacy in Tool-Using LLM Agents](https://mlsanity.com/paper/2606.28061/) (2026)
- [SMCP: Secure Model Context Protocol](https://mlsanity.com/paper/2602.01129/) (2026)
- [HomeGuard: VLM-based Embodied Safeguard for Identifying Contextual Risk in Household Task](https://mlsanity.com/paper/2603.14367/) (2026)
- [Sovereign Agentic Loops: Decoupling AI Reasoning from Execution in Real-World Systems](https://mlsanity.com/paper/2604.22136/) (2026)
- [Training Deliberative Monitors for Black-Box Scheming Detection](https://mlsanity.com/paper/2605.29601/) (2026)
- [ProMAS: Proactive Error Forecasting for Multi-Agent Systems Using Markov Transition Dynamics](https://mlsanity.com/paper/2603.20260/) (2026)
- [Uncertainty Propagation in LLM-Based Systems](https://mlsanity.com/paper/2604.23505/) (2026)
- [Measuring the Unmeasurable: Markov Chain Reliability for LLM Agents](https://mlsanity.com/paper/2604.24579/) (2026)
- [Vertex-Softmax: Tight Transformer Verification via Exact Softmax Optimization](https://mlsanity.com/paper/2605.10974/) (2026)
- [The Saturation Trap and the Subjectivity of Intervention Timing: Why Affect-Based Triggers and LLM Judges Fail to Time Interventions on Autonomous Agents](https://mlsanity.com/paper/2606.04296/) (2026)
- [Don't Solve, Just Compare: Tiny Advisors for Runtime Intervention in LLM Agents](https://mlsanity.com/paper/2608.21027/) (2026)
- [EviDx: Evidence-Aware Active Diagnosis with Scaffolded LLM Agents](https://mlsanity.com/paper/2608.24570/) (2026)
- [Constraining Host-Level Abuse in Self-Hosted Computer-Use Agents via TEE-Backed Isolation](https://mlsanity.com/paper/2605.06393/) (2026)
- [When Routine Chats Turn Toxic: Unintended Long-Term State Poisoning in Personalized Agents](https://mlsanity.com/paper/2605.06731/) (2026)
- [Bistable by Construction: Wall-Clock-Calibrated State Monitors Have No Moment-Detection Regime at Agent Cadence](https://mlsanity.com/paper/2606.19386/) (2026)
- [OpenAgentFlow: Enabling System-Wide Safety Boundaries for Heterogeneous AI Agent Fleets](https://mlsanity.com/paper/2609.00015/) (2026)
- [Representation Transitions Reveal Emerging Safety Risks in Multi-Turn LLM Agents](https://mlsanity.com/paper/2610.00400/) (2026)
- [ATLAS-RTC: Closing the Loop on LLM Agent Output with Token-Level Runtime Control](https://mlsanity.com/paper/2603.27905/) (2026)
- [Formal Verification of Agentic Systems over Operational Data](https://mlsanity.com/paper/2608.03609/) (2026)
- [Symbolic Temporal Supervision of LLM Agents Using Contracts](https://mlsanity.com/paper/2609.18128/) (2026)
- [Epi-Logic: A Conceptual Framework for Epistemic Runtime Control, Schema Validity Checking, and Controlled Accommodation in Autonomous AI Agents](https://mlsanity.com/paper/2609.24755/) (2026)
- [Evaluating System One Models for Agent Security Decisions: Reliability, Calibration, and Selective Automation](https://mlsanity.com/paper/2609.33401/) (2026)
- [Safe Actions Alone Do Not Ensure Safe Agents: Identifying Unfulfilled Obligations with Guard Models](https://mlsanity.com/paper/2610.11773/) (2026)
- [One Monitor Does Not Fit All: Formalism-Agnostic Runtime Verification for Agentic AI](https://mlsanity.com/paper/2610.19818/) (2026)

## Abstract (arXiv)

Large Language Model (LLM) agents increasingly operate across domains such as robotics, virtual assistants, and web automation. However, their stochastic decision-making introduces safety risks that are difficult to anticipate during execution. Existing runtime monitoring frameworks, such as AgentSpec, primarily rely on reactive safety rules that detect violations only when unsafe behavior is imminent or has already occurred, limiting their ability to handle long-horizon dependencies. We present ProbGuard, a proactive runtime monitoring framework for LLM agents that anticipates safety violations through probabilistic risk prediction. ProbGuard abstracts agent executions into symbolic states and learns a Discrete-Time Markov Chain (DTMC) from execution traces to model behavioral dynamics. At runtime, the monitor estimates the probability that execution will remain safe from the current state, and triggers an intervention when this probability falls below a user-defined threshold. To improve robustness, ProbGuard incorporates semantic validity constraints in the abstraction and admits a PAC-style analysis that characterizes the sample complexity required to certify the learned model under standard assumptions. We evaluate ProbGuard in two safety-critical domains: autonomous driving and embodied household agents. Across evaluated scenarios, ProbGuard consistently predicts traffic law violations and collisions in advance, with warnings up to 15.84 seconds at a threshold yielding no false alarms, and up to 38.66 seconds at stricter thresholds. In embodied agent tasks, ProbGuard's re-prompting intervention mode reduces unsafe behavior by 65.37% relative to the unmonitored baseline while retaining 80.4% of the baseline task completion; a stricter halting configuration reduces unsafe behavior by 93.60% at a larger cost in completion.