mlsanityData to 13 Oct 2026

LlamaFirewall: An open source guardrail system for building secure AI agents

arXiv 2505.03574 ↗First posted 6 May 2025cs.CR cs.AIcited by 123

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Large language models (LLMs) have evolved from simple chatbots into autonomous agents capable of performing complex tasks such as editing production code, orchestrating workflows, and taking higher-stakes actions based on untrusted inputs like webpages and emails. These capabilities introduce new security risks that existing security measures, such as model fine-tuning or chatbot-focused guardrails, do not fully address. Given the higher stakes and the absence of deterministic solutions to mitigate these risks, there is a critical need for a real-time guardrail monitor to serve as a final layer of defense, and support system level, use case specific safety policy definition and enforcement. We introduce LlamaFirewall, an open-source security focused guardrail framework designed to serve as a final layer of defense against security risks associated with AI Agents. Our framework mitigates risks such as prompt injection, agent misalignment, and insecure code risks through three powerful guardrails: PromptGuard 2, a universal jailbreak detector that demonstrates clear state of the art performance; Agent Alignment Checks, a chain-of-thought auditor that inspects agent reasoning for prompt injection and goal misalignment, which, while still experimental, shows stronger efficacy at preventing indirect injections in general scenarios than previously proposed approaches; and CodeShield, an online static analysis engine that is both fast and extensible, aimed at preventing the generation of insecure or dangerous code by coding agents. Additionally, we include easy-to-use customizable scanners that make it possible for any developer who can write a regular expression or an LLM prompt to quickly update an agent's security guardrails.

Key facts. Cited by 123 papers in the map (data through 13 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 8 of its 21 references that are in the map. First posted on arXiv 6 May 2025.

Builds on

8 of 21 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Defeating Prompt Injections by Design24 Mar
  2. 2024The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents21 Dec

Other references · 6

  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,171
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 647
  3. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 477
  4. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 357
  5. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 258
  6. 2024Black-Box Adversarial Attacks on LLM-Based Code Completioncited by 14

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 32; 2026: 91.

123 papers.

Sort:

By area:

  1. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 236
  2. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 123
  3. 2025CoT Red-Handed: Stress Testing Chain-of-Thought Monitoringcited by 52
  4. 2026Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Frameworkcited by 52
  5. 2025RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environmentscited by 44
  6. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 43
  7. 2025Measuring Agents in Productioncited by 35
  8. 2026AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocationscited by 35
  9. 2025Building a Foundational Guardrail for General Agentic Systems via Synthetic Datacited by 29
  10. 2026Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agentscited by 29
  11. 2025LLM Agents Should Employ Security Principlescited by 28
  12. 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 28
38 more
  1. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 26
  2. 2025Training LLMs for Honesty via Confessionscited by 21
  3. 2025Towards Enforcing Company Policy Adherence in Agentic Workflowscited by 17
  4. 2025Safety Through Reasoning: An Empirical Study of Reasoning Guardrail Modelscited by 15
  5. 2026SafeHarness: Lifecycle-Integrated Security Architecture for LLM-based Agent Deploymentcited by 15
  6. 2026When Safe Skills Collide: Measuring Compositional Risk in Agent Skill Ecosystemscited by 15
  7. 2026When Benign Inputs Lead to Severe Harms: Eliciting Unsafe Unintended Behaviors of Computer-Use Agentscited by 14
  8. 2025PSG-Agent: Personality-Aware Safety Guardrail for LLM-based Agentscited by 12
  9. 2025Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systemscited by 12
  10. 2026Monitoring Emergent Reward Hacking During Generation via Internal Activationscited by 12
  11. 2026OpenClaw PRISM: A Zero-Fork, Defense-in-Depth Runtime Security Layer for Tool-Augmented LLM Agentscited by 11
  12. 2025HarmonyGuard: Toward Safety and Utility in Web Agents via Adaptive Policy Enhancement and Dual-Objective Optimizationcited by 10
  13. 2026AgentGuardian: Learning Access Control Policies to Govern AI Agent Behaviorcited by 10
  14. 2026When Benchmarks Lie: Evaluating Malicious Prompt Classifiers Under True Distribution Shiftcited by 10
  15. 2025Meta SecAlign: A Secure Foundation LLM Against Prompt Injection Attackscited by 9
  16. 2025Taxonomy, Evaluation and Exploitation of IPI-Centric LLM Agent Defense Frameworkscited by 9
  17. 2026Thought Virus: Viral Misalignment via Subliminal Prompting in Multi-Agent Systemscited by 9
  18. 2025All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Languagecited by 8
  19. 2026Lingering Authority: Revocable Resource-and-Effect Capabilities for Coding Agentscited by 8
  20. 2025When Developer Aid Becomes Security Debt: A Systematic Analysis of Insecure Behaviors in LLM Coding Agentscited by 7
  21. 2025Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chaincited by 6
  22. 2026From Risk Classification to Action Plan Remediation: A Guardrail Feedback Driven Framework for LLM Agentscited by 6
  23. 2026Beyond Single-Use Tokens: Durable Authorization State for Replay-Resistant LLM Agent Actionscited by 6
  24. 2026SHE: Trajectory-driven Safety Harness Evolution for LLM Agentscited by 6
  25. 2025AdvEvo-MARL: Shaping Internalized Safety through Adversarial Co-Evolution in Multi-Agent Reinforcement Learningcited by 5
  26. 2025Trusted AI Agents in the Cloudcited by 5
  27. 2026Aligned, Orthogonal or In-conflict: When can we safely optimize Chain-of-Thought?cited by 5
  28. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 5
  29. 2026Defending against Adaptive Prompt Injection Attacks via Reasoning-enabled Task Alignmentcited by 5
  30. 2025Unsupervised decoding of encoded reasoning using language model interpretabilitycited by 4
  31. 2026Proof-of-Guardrail in AI Agents and What (Not) to Trust from Itcited by 4
  32. 2026HomeGuard: VLM-based Embodied Safeguard for Identifying Contextual Risk in Household Taskcited by 4
  33. 2026Semantic Intent Fragmentation: A Single-Shot Compositional Attack on Multi-Agent AI Pipelinescited by 4
  34. 2026An AI Agent Execution Environment to Safeguard User Datacited by 4
  35. 2026Your Agentic LLMs Secretly Encode Indirect Prompt-Injection Exposure in Hidden Statescited by 4
  36. 2025MCP Safety Training: Learning to Refuse Falsely Benign MCP Exploits using Improved Preference Alignmentcited by 3
  37. 2025When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?cited by 3
  38. 2026MCPShield: A Security Cognition Layer for Adaptive Trust Calibration in Model Context Protocol Agentscited by 3

50 most cited of 123 All 123 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ASTRA: Agentic Steerability and Risk Assessment Framework
  2. 2026Toward Secure AI-Powered Penetration Testing Agents: Security Threats, Guardrails, and Architectural Perspectives
  3. 2026Behind the Refusal: Determining Guardrail Activation via Behavioral Monitoring
  4. 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection
  5. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning
  6. 2026TraceSafe: A Systematic Assessment of LLM Guardrails on Multi-Step Tool-Calling Trajectories
  7. 2026DreamGuard: Efficient Runtime Guardrail for LLM Agents via Risk-Aware World Model
  8. 2025OpenGuardrails: A Configurable, Unified, and Scalable Guardrails Platform for Large Language Models