Building a Foundational Guardrail for General Agentic Systems via Synthetic Data
On the map: LLM agents and tool use → Agent threats, guardrails and governance
also in 2 more
Abstract (arXiv)
While LLM agents can plan multi-step tasks, intervening at the planning stage-before any action is executed-is often the safest way to prevent harm, since certain risks can lead to severe consequences once carried out. However, existing guardrails mostly operate post-execution, which is difficult to scale and leaves little room for controllable supervision at the plan level. To address this challenge, we highlight three critical gaps in current research: data gap, model gap, and evaluation gap. To close the data gap, we introduce AuraGen, a controllable engine that (i) synthesizes benign trajectories, (ii) injects category-labeled risks with calibrated difficulty, and (iii) filters outputs via an automated reward model, producing large and reliable corpora for pre-execution safety. To close the guardian model gap, we propose a foundational guardrail Safiron, combining a cross-planner adapter with a compact guardian model. The adapter unifies different input formats, while Safiron flags risky cases, assigns risk types, and generates rationales; trained in two stages with a broadly explored data recipe, Safiron achieves robust transfer across settings. To close the evaluation gap, we release Pre-Exec Bench, a realistic benchmark covering diverse tools and branching trajectories, which measures detection, fine-grained categorization, explanation, and cross-planner generalization in human-verified scenarios. Extensive experiments demonstrate consistent gains of the proposed guardrail over strong baselines on Pre-Exec Bench, and ablations further distill actionable practices, providing a practical template for safer agentic systems.
Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 52 of its 71 references that are in the map. First posted on arXiv 10 Oct 2025.
Builds on
52 of 71 references.
Closest ancestors · 18
Same direction, newest first
- 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety8 Jul
- 2025Enhancing LLM Agent Safety via Causal Influence Prompting1 Jul
- 2025AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agents31 May
- 2025SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents29 May
- 2025GUARDIAN: Safeguarding LLM Multi-Agent Collaborations with Temporal Graph Modeling25 May
- 2025LlamaFirewall: An open source guardrail system for building secure AI agents6 May
- 2025Prompt Injection Attack to Tool Selection in LLM Agents28 Apr
- 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment22 Apr
- 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning26 Mar
- 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents24 Mar
- 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection17 Feb
- 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks12 Feb
6 more
- 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents19 Dec
- 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents17 Dec
- 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning13 Jun
- 2024TrustAgent: Towards Safe and Trustworthy LLM-based Agents2 Feb
- 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agents18 Jan
- 2023Evil Geniuses: Delving into the Safety of LLM-based Agents20 Nov
Other references · 34
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
- 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
- 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
- 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
- 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
- 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
- 2024Understanding the planning of LLM agents: A surveycited by 414
- 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
22 more
- 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
- 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
- 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 224
- 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
- 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
- 2023Simple synthetic data reduces sycophancy in large language modelscited by 174
- 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
- 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 121
- 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
- 2024Safeguarding Large Language Models: A Surveycited by 73
- 2024Risks of AI Scientists: Prioritizing Safeguarding Over Autonomycited by 49
- 2025On the Trustworthiness of Generative Foundation Models: Guideline, Assessment, and Perspectivecited by 39
- 2025Safety Guardrails for LLM-Enabled Robotscited by 30
- 2023Bergeron: Combating Adversarial Attacks through a Conscience-Based Alignment Frameworkcited by 29
- 2025MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Sciencecited by 22
- 2024SyntheT2C: Generating Synthetic Data for Fine-Tuning Large Language Models on the Text2Cypher Taskcited by 19
- 2025Oasis: One Image is All You Need for Multimodal Instruction Data Synthesiscited by 13
- 2025MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generationcited by 10
- 2023Silent Guardian: Protecting Text from Malicious Exploitation by Large Language Modelscited by 8
- 2024Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settingscited by 8
- 2024A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detectioncited by 3
- 2025ChemOrch: Empowering LLMs with Chemical Intelligence via Synthetic Instructionscited by 3
Built on by
Citing papers per year: 2025: 1; 2026: 25.
26 papers.
Sort:
By area:
- 2026ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedbackcited by 64
- 2026ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosiscited by 30
- 2026SkillGen: Verified Inference-Time Agent Skill Synthesiscited by 29
- 2026Before the Tool Call: Deterministic Pre-Action Authorization for Autonomous AI Agentscited by 12
- 2026MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agentscited by 9
- 2026When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agentscited by 8
- 2026AIR: Improving Agent Safety through Incident Responsecited by 8
- 2026ProbeLLM: Automating Principled Diagnosis of LLM Failurescited by 7
- 2026From Governance Norms to Enforceable Controls: A Layered Translation Method for Runtime Guardrails in Agentic AIcited by 7
- 2026AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Securitycited by 7
- 2026From Risk Classification to Action Plan Remediation: A Guardrail Feedback Driven Framework for LLM Agentscited by 5
- 2026SafeMCP: Proactive Power Regulation for LLM Agent Defense via Environment-Grounded Look-Ahead Reasoningcited by 3
14 more
- 2026Do LLMs Know What Is Private Internally? Probing and Steering Contextual Privacy Norms in Large Language Model Representationscited by 2
- 2026Getting Better at Working With You: Compiling User Corrections into Runtime Enforcement for Coding Agentscited by 2
- 2025Can LLM Infer Risk Information From MCP Server System Logs?cited by 1
- 2026Guardian-as-an-Advisor: Advancing Next-Generation Guardian Models for Trustworthy LLMscited by 1
- 2026TRACE: Trajectory Risk-Aware Compression for Long-Horizon Agent Safetycited by 1
- 2026Agent Skills Should Go Beyond Text: The Case for Visual Skillscited by 1
- 2026GuardianAgentBench: Where Agents Fail and How to Guard Themcited by 1
- 2026Reward Hacking Challenges Oversight of Autonomous Research Agentscited by 1
- 2026SAFETY SENTRY: Context-Aware Human Intervention via EXECUTE-ASK-REFUSE Routingcited by 0
- 2026JANUS: Foreseeing Latent Risk for Long-Horizon Agent Safetycited by 0
- 2026Delegation Intelligence in Deep Search: A Controllable Framework for Disentangled Capability Diagnosiscited by 0
- 2026MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory Learningcited by 0
- 2026PROACT-Agent: Progressive Runtime Oversight and Active Circuit-breaking for Real-Time Safetycited by 0
- 2026SEAD: A State-Based Perspective on Attack and Defense in Tool-Using Agentscited by 0
- 2026SEAD: A State-Based Perspective on Attack and Defense in Tool-Using Agents28 Sep 2026
- 2026PROACT-Agent: Progressive Runtime Oversight and Active Circuit-breaking for Real-Time Safety28 Sep 2026
- 2026Reward Hacking Challenges Oversight of Autonomous Research Agents23 Sep 2026
- 2026MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory Learning19 Sep 2026
- 2026Delegation Intelligence in Deep Search: A Controllable Framework for Disentangled Capability Diagnosis26 Jul 2026
- 2026GuardianAgentBench: Where Agents Fail and How to Guard Them23 Jul 2026
- 2026JANUS: Foreseeing Latent Risk for Long-Horizon Agent Safety22 Jul 2026
- 2026SAFETY SENTRY: Context-Aware Human Intervention via EXECUTE-ASK-REFUSE Routing15 Jul 2026
- 2026Getting Better at Working With You: Compiling User Corrections into Runtime Enforcement for Coding Agents11 Jun 2026
- 2026From Risk Classification to Action Plan Remediation: A Guardrail Feedback Driven Framework for LLM Agents4 Jun 2026
- 2026SafeMCP: Proactive Power Regulation for LLM Agent Defense via Environment-Grounded Look-Ahead Reasoning1 Jun 2026
- 2026Agent Skills Should Go Beyond Text: The Case for Visual Skills31 May 2026
14 more
- 2026TRACE: Trajectory Risk-Aware Compression for Long-Horizon Agent Safety30 May 2026
- 2026AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Security28 May 2026
- 2026MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents18 May 2026
- 2026SkillGen: Verified Inference-Time Agent Skill Synthesis9 May 2026
- 2026Guardian-as-an-Advisor: Advancing Next-Generation Guardian Models for Trustworthy LLMs8 Apr 2026
- 2026From Governance Norms to Enforceable Controls: A Layered Translation Method for Runtime Guardrails in Agentic AI6 Apr 2026
- 2026ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis2 Apr 2026
- 2026Do LLMs Know What Is Private Internally? Probing and Steering Contextual Privacy Norms in Large Language Model Representations31 Mar 2026
- 2026Before the Tool Call: Deterministic Pre-Action Authorization for Autonomous AI Agents21 Mar 2026
- 2026ProbeLLM: Automating Principled Diagnosis of LLM Failures13 Feb 2026
- 2026AIR: Improving Agent Safety through Incident Response12 Feb 2026
- 2026When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents9 Feb 2026
- 2026ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedback15 Jan 2026
- 2025Can LLM Infer Risk Information From MCP Server System Logs?8 Nov 2025
- 2026ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedbackcited by 64
- 2026ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosiscited by 30
- 2026SkillGen: Verified Inference-Time Agent Skill Synthesiscited by 29
- 2026MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agentscited by 9
- 2026When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agentscited by 8
- 2026AIR: Improving Agent Safety through Incident Responsecited by 8
- 2026From Governance Norms to Enforceable Controls: A Layered Translation Method for Runtime Guardrails in Agentic AIcited by 7
- 2026AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Securitycited by 7
- 2026From Risk Classification to Action Plan Remediation: A Guardrail Feedback Driven Framework for LLM Agentscited by 5
- 2026Getting Better at Working With You: Compiling User Corrections into Runtime Enforcement for Coding Agentscited by 2
- 2026TRACE: Trajectory Risk-Aware Compression for Long-Horizon Agent Safetycited by 1
- 2026GuardianAgentBench: Where Agents Fail and How to Guard Themcited by 1
12 most cited of 17 All 17 (JSON)
- 2026SEAD: A State-Based Perspective on Attack and Defense in Tool-Using Agents28 Sep 2026
- 2026PROACT-Agent: Progressive Runtime Oversight and Active Circuit-breaking for Real-Time Safety28 Sep 2026
- 2026Reward Hacking Challenges Oversight of Autonomous Research Agents23 Sep 2026
- 2026MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory Learning19 Sep 2026
- 2026GuardianAgentBench: Where Agents Fail and How to Guard Them23 Jul 2026
- 2026SAFETY SENTRY: Context-Aware Human Intervention via EXECUTE-ASK-REFUSE Routing15 Jul 2026
- 2026Getting Better at Working With You: Compiling User Corrections into Runtime Enforcement for Coding Agents11 Jun 2026
- 2026From Risk Classification to Action Plan Remediation: A Guardrail Feedback Driven Framework for LLM Agents4 Jun 2026
- 2026TRACE: Trajectory Risk-Aware Compression for Long-Horizon Agent Safety30 May 2026
- 2026AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Security28 May 2026
- 2026MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents18 May 2026
- 2026SkillGen: Verified Inference-Time Agent Skill Synthesis9 May 2026
12 newest of 17 All 17 (JSON)
- 2026Before the Tool Call: Deterministic Pre-Action Authorization for Autonomous AI Agentscited by 12
- 2026ProbeLLM: Automating Principled Diagnosis of LLM Failurescited by 7
- 2026Do LLMs Know What Is Private Internally? Probing and Steering Contextual Privacy Norms in Large Language Model Representationscited by 2
- 2026Guardian-as-an-Advisor: Advancing Next-Generation Guardian Models for Trustworthy LLMscited by 1
- 2026Agent Skills Should Go Beyond Text: The Case for Visual Skillscited by 1
- 2026JANUS: Foreseeing Latent Risk for Long-Horizon Agent Safetycited by 0
- 2026Delegation Intelligence in Deep Search: A Controllable Framework for Disentangled Capability Diagnosiscited by 0
- 2026Delegation Intelligence in Deep Search: A Controllable Framework for Disentangled Capability Diagnosis26 Jul 2026
- 2026JANUS: Foreseeing Latent Risk for Long-Horizon Agent Safety22 Jul 2026
- 2026Agent Skills Should Go Beyond Text: The Case for Visual Skills31 May 2026
- 2026Guardian-as-an-Advisor: Advancing Next-Generation Guardian Models for Trustworthy LLMs8 Apr 2026
- 2026Do LLMs Know What Is Private Internally? Probing and Steering Contextual Privacy Norms in Large Language Model Representations31 Mar 2026
- 2026Before the Tool Call: Deterministic Pre-Action Authorization for Autonomous AI Agents21 Mar 2026
- 2026ProbeLLM: Automating Principled Diagnosis of LLM Failures13 Feb 2026
Similar, not cited
Close in wording, no citation either way
- 2026DreamGuard: Efficient Runtime Guardrail for LLM Agents via Risk-Aware World Model
- 2026EMBGuard: Constructing Hazard-Aware Guardrails for Safe Planning in Embodied Agents
- 2026AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Security
- 2026StepGuard: Learning Step-Level Guardrails with Scalable Supervision and Safety-Utility Balancing
- 2026Yesterday's Shield, Today's Spear: A Self-Evolving Safety Guardrail in Production
- 2026PreAct-Bench: Benchmarking Predictive Monitoring in LLMs
- 2025SafeMind: Benchmarking and Mitigating Safety Risks in Embodied LLM Agents
- 2026TRACES: Proactive Safety Auditing for Multi-Turn LLM Agents via Trajectory-State Modeling