Reframing LLM Agent Security as an Agent-Human Interaction Problem
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
We argue that LLM agent security is fundamentally an agent-human interaction (AHI) problem, not a purely algorithmic one. To substantiate this position, we conduct a systematic analysis of 59 academic papers, 21 production agent systems, and 26 security plugins as of April 2026. Our analysis reveals a striking pattern: the three widely deployed human-centric security mechanisms (policy specification, runtime approval, and scope configuration) dominate industry practice, each adopted by at least 14 of 21 systems (14, 15, and 16, respectively), while the categories most heavily studied in academia (intent anchoring and trust labeling) see zero production deployment. Yet current human participation mechanisms are far from satisfactory: they suffer from a fundamental trade-off between cognitive burden and security guarantees, leaving users caught between approval fatigue and uncontrolled agent autonomy. We make three contributions. First, through a systematic comparison of LLM-based and human-based intent alignment, we argue that human participation in agent security decisions is indispensable given current capabilities. Second, we quantify a pronounced industry-academia mismatch: the security mechanisms that practitioners actually deploy receive scant research attention, while the approaches that researchers favor remain undeployed. Third, we propose a three-direction research agenda and call for AHI security to be recognized as a first-class research citizen, one that demands its own design principles, evaluation methods, and theoretical foundations.
Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 38 of its 64 references that are in the map. First posted on arXiv 23 May 2026.
Builds on
38 of 64 references.
Closest ancestors · 19
Same direction, newest first
- 2026Architecting Secure AI Agents: Perspectives on System-Level Defenses Against Indirect Prompt Injection Attacks31 Mar
- 2026Agent-Sentry: Bounding LLM Agents via Execution Provenance24 Mar
- 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysis11 Feb
- 2026Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Framework17 Jan
- 2026Towards Verifiably Safe Tool Use for LLM Agents12 Jan
- 2025Fault-Tolerant Sandboxing for AI Coding Agents: A Transactional Approach to Safe Autonomous Execution14 Dec
- 2025Secure and Efficient Access Control for Computer-Use Agents via Context Space26 Sep
- 2025LLMZ+: Contextual Prompt Whitelist Principles for Agentic LLMs23 Sep
- 2025AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injection2 Aug
- 2025DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agents13 Jun
- 2025SAFEFLOW: A Principled Protocol for Trustworthy and Transactional Autonomous Agent Systems9 Jun
- 2025Defeating Prompt Injections by Design24 Mar
7 more
- 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents24 Mar
- 2025Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agents17 Mar
- 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents27 Feb
- 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection17 Feb
- 2025RTBAS: Defending LLM Agents Against Prompt Injection and Privacy Leakage13 Feb
- 2024The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents21 Dec
- 2024IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systems8 Mar
Other references · 19
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2024Understanding the planning of LLM agents: A surveycited by 414
- 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
- 2025Securing AI Agents with Information-Flow Controlcited by 108
- 2024System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspectivecited by 91
- 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 90
- 2025Contextual Agent Security: A Policy for Every Purposecited by 55
- 2025ACE: A Security Architecture for LLM-Integrated App Systemscited by 52
- 2025What Prompts Don't Say: Understanding and Managing Underspecification in LLM Promptscited by 32
- 2025Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineeringcited by 28
- 2026Don't Let the Claw Grip Your Hand: A Security Analysis and Defense Framework for OpenClawcited by 26
7 more
- 2024Permissive Information-Flow Analysis for Large Language Modelscited by 22
- 2025Towards Automating Data Access Permissions in AI Agentscited by 20
- 2024FATH: Authentication-based Test-time Defense against Indirect Prompt Injection Attackscited by 19
- 2024Contextualized Evaluations: Judging Language Model Responses to Underspecified Queriescited by 12
- 2026AgentGuardian: Learning Access Control Policies to Govern AI Agent Behaviorcited by 9
- 2025On the Regulatory Potential of User Interfaces for AI Agent Governancecited by 3
- 2023Stuck in the Permissions With You: Developer & End-User Perspectives on App Permissions & Their Privacy Ramificationscited by 2
Built on by
Citing papers per year: 2026: 8.
8 papers.
- 2026Oversight Has a Capacity: Calibrating Agent Guards to a Subjective, Fatiguing Humancited by 3
- 2026Janus: a Playground for User-Involved Agentic Permission Managementcited by 2
- 2026How Agents Ask for Permission: User Permissions for AI Agents, from Interfaces to Enforcementcited by 2
- 2026One Goal, Many Commands: Characterizing Denylist Fragility in AI Agentscited by 1
- 2026ERPBench: A State-Grounded Evaluation Paradigm for Computer-Use Agents in Enterprise Softwarecited by 1
- 2026The Balkanization of Execution-Security Research for AI Coding Agents: Isolation, Access Control, and Time-of-Check-to-Time-of-Use Vulnerabilitiescited by 0
- 2026Do User-Authored Permission Policies Improve Protection Against AI Agent Overreach?cited by 0
- 2026When the Agent Becomes the Kernel: A Systematization of Security on the Path to AI-Native Operating Systemscited by 0
- 2026When the Agent Becomes the Kernel: A Systematization of Security on the Path to AI-Native Operating Systems20 Sep 2026
- 2026ERPBench: A State-Grounded Evaluation Paradigm for Computer-Use Agents in Enterprise Software15 Sep 2026
- 2026Do User-Authored Permission Policies Improve Protection Against AI Agent Overreach?27 Aug 2026
- 2026How Agents Ask for Permission: User Permissions for AI Agents, from Interfaces to Enforcement15 Jul 2026
- 2026The Balkanization of Execution-Security Research for AI Coding Agents: Isolation, Access Control, and Time-of-Check-to-Time-of-Use Vulnerabilities7 Jul 2026
- 2026Janus: a Playground for User-Involved Agentic Permission Management1 Jul 2026
- 2026One Goal, Many Commands: Characterizing Denylist Fragility in AI Agents14 Jun 2026
- 2026Oversight Has a Capacity: Calibrating Agent Guards to a Subjective, Fatiguing Human8 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2026LLM-Based Agents for Software and Systems Security: Approaches, Applications, and Assessment
- 2026Connecting the Dots in Agentic AI Security: A Cross-Dimensional Threat Taxonomy, Evaluation Maturity, and Open Challenges
- 2025TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systems
- 2026From Human-Human Collaboration to Human-Agent Collaboration: A Vision, Design Philosophy, and an Empirical Framework for Achieving Successful Partnerships Between Humans and LLM Agents
- 2026Delegation Without Trust: An Empirical Gap Analysis of Identity, Authorization, and Runtime Governance in Multi-Agent LLM Systems
- 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation
- 2026How Well Can LLM Agents Simulate End-User Security and Privacy Attitudes and Behaviors?
- 2025Seven Security Challenges in Cross-domain Multi-agent LLM Systems