Security and Privacy in Agentic AI: Grand Challenges and Future Directions
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
We present key challenges and future research directions in the security and privacy of agentic AI, based on a horizon-scanning exercise that brought together thirty leading international experts from academia, industry, and government to engage in focused discussions and collaborative exercises on the emerging risks associated with the growing agency of AI.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 47 of its 90 references that are in the map. First posted on arXiv 7 Jul 2026.
Builds on
47 of 90 references.
Closest ancestors · 3
Same direction, newest first
- 2026AgentSCOPE: Evaluating Contextual Privacy Across Agentic Workflows5 Mar
- 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
- 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training10 Jan
Other references · 44
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
- 2023Towards Understanding Sycophancy in Language Modelscited by 988
- 2016Inherent Trade-Offs in the Fair Determination of Risk Scorescited by 582
- 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
- 2018AI safety via debatecited by 384
- 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 380
- 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 328
- 2025Small Language Models are the Future of Agentic AIcited by 254
32 more
- 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
- 2019A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cyclecited by 192
- 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
- 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 161
- 2025Multi-Agent Risks from Advanced AIcited by 150
- 2023Harms from Increasingly Agentic Algorithmic Systemscited by 134
- 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 132
- 2024Cognitive Bias in Decision-Making with LLMscited by 104
- 2024Large Language Models Must Be Taught to Know What They Don't Knowcited by 99
- 2023Understanding accountability in algorithmic supply chainscited by 39
- 2025Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistenciescited by 39
- 2024m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Taskscited by 38
- 2024On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectivescited by 28
- 2021Outlining Traceability: A Principle for Operationalizing Accountability in Computing Systemscited by 26
- 2020Bias and Discrimination in AI: a cross-disciplinary perspectivecited by 24
- 2021An Objective Metric for Explainable AI: How and Why to Estimate the Degree of Explainabilitycited by 15
- 2025Malicious LLM-Based Conversational AI Makes Users Reveal Personal Informationcited by 10
- 2026Audit Trails for Accountability in Large Language Modelscited by 10
- 2022Detecting Outdated Code Element References in Software Repository Documentationcited by 5
- 2025Large Language Models for In-File Vulnerability Localization Can Be "Lost in the End"cited by 5
- 2025Accountability Capture: How Record-Keeping to Support AI Transparency and Accountability (Re)shapes Algorithmic Oversightcited by 5
- 2026AI Agents Under EU Lawcited by 5
- 2025Can Global XAI Methods Reveal Injected Behaviours in LLMs? SHAP vs Rule Extraction vs RuleSHAPcited by 3
- 2023An Empirical Study on Compliance with Ranking Transparency in the Software Documentation of EU Online Platformscited by 2
- 2025Is General-Purpose AI Reasoning Sensitive to Data-Induced Cognitive Biases? Dynamic Benchmarking on Typical Software Engineering Dilemmascited by 2
- 2026Privacy in Human-AI Romantic Relationships: Concerns, Boundaries, and Agencycited by 2
- 2026Neuron-Anchored Rule Extraction for Large Language Models via Contrastive Hierarchical Ablationcited by 2
- 2022Harpocrates: Privacy-Preserving and Immutable Audit Log for Sensitive Data Operationscited by 1
- 2024On Cryptographic Mechanisms for the Selective Disclosure of Verifiable Credentialscited by 1
- 2026Illocutionary Explanation Planning for Source-Faithful Explanations in Retrieval-Augmented Language Modelscited by 1
- 2026Assessing Model-Agnostic XAI Methods against EU AI Act Explainability Requirementscited by 1
- 2026Mitigating Prompt-Induced Cognitive Biases in General-Purpose AI for Software Engineeringcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Security, privacy, and agentic AI in a regulatory view: From definitions and distinctions to provisions and reflections
- 2026A Security Analysis of Long-Horizon Agentic AI Systems: Threats, Evaluation, and Framework Development
- 2026Connecting the Dots in Agentic AI Security: A Cross-Dimensional Threat Taxonomy, Evaluation Maturity, and Open Challenges
- 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey
- 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
- 2024Security Threats in Agentic AI System
- 2026AgentSocialBench: Evaluating Privacy Risks in Human-Centered Agentic Social Networks
- 2025AgentCrypt: Advancing Privacy and (Secure) Computation in AI Agent Collaboration