Trustworthy Agentic AI Requires Deterministic Architectural Boundaries
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
Current agentic AI architectures are fundamentally incompatible with the security and epistemological requirements of high-stakes scientific workflows. The problem is not inadequate alignment or insufficient guardrails, it is architectural: autoregressive language models process all tokens uniformly, making deterministic command--data separation unattainable through training alone. We argue that deterministic, architectural enforcement, not probabilistic learned behavior, is a necessary condition for trustworthy AI-assisted science. We introduce the Trinity Defense Architecture, which enforces security through three mechanisms: action governance via a finite action calculus with reference-monitor enforcement, information-flow control via mandatory access labels preventing cross-scope leakage, and privilege separation isolating perception from execution. We show that without unforgeable provenance and deterministic mediation, the ``Lethal Trifecta'' (untrusted inputs, privileged data access, external action capability) turns authorization security into an exploit-discovery problem: training-based defenses may reduce empirical attack rates but cannot provide deterministic guarantees. The ML community must recognize that alignment is insufficient for authorization security, and that architectural mediation is required before agentic AI can be safely deployed in consequential scientific domains.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 25 of its 58 references that are in the map. First posted on arXiv 10 Feb 2026.
Builds on
25 of 58 references.
Closest ancestors · 2
Same direction, newest first
- 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
- 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep
Other references · 23
- 2017Attention Is All You Needcited by 59,932
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
- 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
11 more
- 2022Red Teaming Language Models with Language Modelscited by 983
- 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
- 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
- 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
- 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
- 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 260
- 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
- 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 165
- 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
- 2023Evaluating Language-Model Agents on Realistic Autonomous Taskscited by 94
- 2024Cedar: A New Language for Expressive, Fast, Safe, and Analyzable Authorization (Extended Version)cited by 31
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026Before the Tool Call: Deterministic Pre-Action Authorization for Autonomous AI Agentscited by 12
- 2026Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agentscited by 2
- 2026Toward Self-Evolution-Ready Workflow Harnesses: A Reversible Migration Path and Convertibility Taxonomy for Expert LLM Pipelinescited by 0
- 2026Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agents25 Jun 2026
- 2026Toward Self-Evolution-Ready Workflow Harnesses: A Reversible Migration Path and Convertibility Taxonomy for Expert LLM Pipelines15 Jun 2026
- 2026Before the Tool Call: Deterministic Pre-Action Authorization for Autonomous AI Agents21 Mar 2026
Similar, not cited
Close in wording, no citation either way
- 2026Trustworthy Agentic AI: A Comprehensive Cybersecurity and Systems Survey on Threat Landscapes, Defense Architectures, and Open Challenges
- 2026AgentKernel: The Trust-Native Agentic Operating System
- 2026Verifiable Agentic Infrastructure: Proof-Derived Authorization for Sovereign AI Systems
- 2026A Policy Algebra for Trust-Preserving Agentic AI Execution
- 2026Authenticated Workflows: A Systems Approach to Protecting Agentic AI
- 2026From Craft to Kernel: A Governance-First Execution Architecture and Semantic ISA for Agentic Computers
- 2025Safe, Untrusted, "Proof-Carrying" AI Agents: toward the agentic lakehouse
- 2026Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security