# SentinelAgent: Graph-based Anomaly Detection in Multi-Agent Systems

arXiv: https://arxiv.org/abs/2505.24201 · HTML: https://mlsanity.com/paper/2505.24201/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 5 of its 43 references that are in the map.
- First posted on arXiv 30 May 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)
- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 5 of 5)

- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents](https://mlsanity.com/paper/2306.03314/) (2023)
- [Frontier Models are Capable of In-context Scheming](https://mlsanity.com/paper/2412.04984/) (2024)
- [Multi-Agent Systems Execute Arbitrary Malicious Code](https://mlsanity.com/paper/2503.12188/) (2025)
- [Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysis](https://mlsanity.com/paper/2502.20383/) (2025)

## Built on by (most cited first, 22 of 22)

- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [AgentBound: Securing Execution Boundaries of AI Agents](https://mlsanity.com/paper/2510.21236/) (2025)
- [From Spark to Fire: Modeling and Mitigating Error Cascades in LLM-Based Multi-Agent Collaboration](https://mlsanity.com/paper/2603.04474/) (2026)
- [Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systems](https://mlsanity.com/paper/2510.17276/) (2025)
- [SoK: Trust-Authorization Mismatch in LLM Agent Interactions](https://mlsanity.com/paper/2512.06914/) (2025)
- [Detecting Silent Failures in Multi-Agentic AI Trajectories](https://mlsanity.com/paper/2511.04032/) (2025)
- [Detecting Multi-Agent Collusion Through Multi-Agent Interpretability](https://mlsanity.com/paper/2604.01151/) (2026)
- [Self-Evolving Agents as Dynamic Graph Transformation: A Survey and New Perspective](https://mlsanity.com/paper/2608.18104/) (2026)
- [CatchBench: When Can an Agent Failure Be Caught?](https://mlsanity.com/paper/2608.22808/) (2026)
- [Automata from Agent Traces: Failure and Next-Step Prediction](https://mlsanity.com/paper/2608.23670/) (2026)
- [AegisMCP: Online Graph Intrusion Detection for Tool-Augmented LLMs on Edge Devices](https://mlsanity.com/paper/2510.19462/) (2025)
- [LLM-enabled Applications Require System-Level Threat Monitoring](https://mlsanity.com/paper/2602.19844/) (2026)
- [GAMMAF: A Common Framework for Graph-Based Anomaly Monitoring Benchmarking in LLM Multi-Agent Systems](https://mlsanity.com/paper/2604.24477/) (2026)
- [DCFA: Dual-view Causal-inspired Attribution for Failure Reasoning in LLM-based Multi-agent Systems](https://mlsanity.com/paper/2609.04749/) (2026)
- [Challenges and Future Directions in Agentic Reverse Engineering Systems](https://mlsanity.com/paper/2604.14317/) (2026)
- [Content-Aware Attack Detection in LLM Agent Tool-Call Traffic: An Empirical Study of Features, Architectures, and Evaluation Protocols](https://mlsanity.com/paper/2605.11053/) (2026)
- [SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems](https://mlsanity.com/paper/2609.00595/) (2026)
- [Unsupervised Cycle Detection in Agentic Applications](https://mlsanity.com/paper/2511.10650/) (2025)
- [Agent System Operations: Categorization, Challenges, and Future Directions](https://mlsanity.com/paper/2606.01581/) (2026)
- [MESA: Prioritizing Vulnerable Communication Channels for Securing Multi-Agent Systems](https://mlsanity.com/paper/2606.30602/) (2026)
- [Runtime Uncertainty Monitoring for LLM-Based Multi-Agent Systems Using Bayesian Networks](https://mlsanity.com/paper/2607.25877/) (2026)
- [Beyond Single-Model Injection: A Threat Model and Defense Architecture for Prompt Injection in Multi-Agent Systems](https://mlsanity.com/paper/2609.22949/) (2026)

## Abstract (arXiv)

The rise of large language model (LLM)-based multi-agent systems (MAS) introduces new security and reliability challenges. While these systems show great promise in decomposing and coordinating complex tasks, they also face multi-faceted risks across prompt manipulation, unsafe tool usage, and emergent agent miscoordination. Existing guardrail mechanisms offer only partial protection, primarily at the input-output level, and fall short in addressing systemic or multi-point failures in MAS. In this work, we present a system-level anomaly detection framework tailored for MAS, integrating structural modeling with runtime behavioral oversight. Our approach consists of two components. First, we propose a graph-based framework that models agent interactions as dynamic execution graphs, enabling semantic anomaly detection at node, edge, and path levels. Second, we introduce a pluggable SentinelAgent, an LLM-powered oversight agent that observes, analyzes, and intervenes in MAS execution based on security policies and contextual reasoning. By bridging abstract detection logic with actionable enforcement, our method detects not only single-point faults and prompt injections but also multi-agent collusion and latent exploit paths. We validate our framework through two case studies, including an email assistant and Microsoft's Magentic-One system, demonstrating its ability to detect covert risks and provide explainable root-cause attribution. Our work lays the foundation for more trustworthy, monitorable, and secure agent-based AI ecosystems.