# A Safety and Security Framework for Real-World Agentic Systems arXiv: https://arxiv.org/abs/2511.21990 ยท HTML: https://mlsanity.com/paper/2511.21990/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. - Builds on 11 of its 30 references that are in the map. - First posted on arXiv 27 Nov 2025. ## Placed in - LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/) ## Builds on (most cited first, 11 of 11) - [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023) - [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023) - [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023) - [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023) - [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024) - [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024) - [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://mlsanity.com/paper/2312.02119/) (2023) - [Against The Achilles' Heel: A Survey on Red Teaming for Generative Models](https://mlsanity.com/paper/2404.00629/) (2024) - [garak: A Framework for Security Probing Large Language Models](https://mlsanity.com/paper/2406.11036/) (2024) - [An Approach to Technical AGI Safety and Security](https://mlsanity.com/paper/2504.01849/) (2025) - [Lessons from Defending Gemini Against Indirect Prompt Injections](https://mlsanity.com/paper/2505.14534/) (2025) ## Built on by (most cited first, 13 of 13) - [AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Security](https://mlsanity.com/paper/2605.29801/) (2026) - [StepGuard: Learning Step-Level Guardrails with Scalable Supervision and Safety-Utility Balancing](https://mlsanity.com/paper/2608.24777/) (2026) - [The Authorization-Execution Gap Is a Major Safety and Security Problem in Open-World Agents](https://mlsanity.com/paper/2605.11003/) (2026) - [Self-Evolving Agents as Dynamic Graph Transformation: A Survey and New Perspective](https://mlsanity.com/paper/2608.18104/) (2026) - [MCP-DPT: A Defense-Placement Taxonomy and Coverage Analysis for Model Context Protocol Security](https://mlsanity.com/paper/2604.07551/) (2026) - [When Child Inherits: Modeling and Exploiting Subagent Spawn in Multi-Agent Networks](https://mlsanity.com/paper/2605.08460/) (2026) - [S3: Improving Agent Safety through Multi-Stage Defense](https://mlsanity.com/paper/2608.02683/) (2026) - [Signals in the Noise: Open Source Intelligence (OSINT) for AI Loss of Control Detection](https://mlsanity.com/paper/2606.20610/) (2026) - [RIFT-Bench: Dynamic Red-teaming For Agentic AI Systems](https://mlsanity.com/paper/2606.23927/) (2026) - [Unaccountable Delegation, Fading Skills: Mapping the Risks of Workplace AI Agents](https://mlsanity.com/paper/2608.08601/) (2026) - [Testing and Evaluation of Agentic AI Systems In Military Command and Control](https://mlsanity.com/paper/2608.20597/) (2026) - [PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety](https://mlsanity.com/paper/2609.28197/) (2026) - [PROACT-Agent: Progressive Runtime Oversight and Active Circuit-breaking for Real-Time Safety](https://mlsanity.com/paper/2609.34415/) (2026) ## Abstract (arXiv) This paper introduces a dynamic and actionable framework for securing agentic AI systems in enterprise deployment. We contend that safety and security are not merely fixed attributes of individual models but also emergent properties arising from the dynamic interactions among models, orchestrators, tools, and data within their operating environments. We propose a new way of identification of novel agentic risks through the lens of user safety. Although, for traditional LLMs and agentic models in isolation, safety and security has a clear separation, through the lens of safety in agentic systems, they appear to be connected. Building on this foundation, we define an operational agentic risk taxonomy that unifies traditional safety and security concerns with novel, uniquely agentic risks, including tool misuse, cascading action chains, and unintended control amplification among others. At the core of our approach is a dynamic agentic safety and security framework that operationalizes contextual agentic risk management by using auxiliary AI models and agents, with human oversight, to assist in contextual risk discovery, evaluation, and mitigation. We further address one of the most challenging aspects of safety and security of agentic systems: risk discovery through sandboxed, AI-driven red teaming. We demonstrate the framework effectiveness through a detailed case study of NVIDIA flagship agentic research assistant, AI-Q Research Assistant, showcasing practical, end-to-end safety and security evaluations in complex, enterprise-grade agentic workflows. This risk discovery phase finds novel agentic risks that are then contextually mitigated. We also release the dataset from our case study, containing traces of over 10,000 realistic attack and defense executions of the agentic workflow to help advance research in agentic safety.