Whispers in the Machine: Confidentiality in Agentic Systems
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
Large language model (LLM)-based agents combine LLMs with external tools to automate tasks such as scheduling meetings, managing documents, or booking travel. While these integrations unlock powerful capabilities, they also create new and more severe attack surfaces. In particular, prompt injection attacks become far more dangerous in the agentic setting: malicious instructions embedded in connected services can misdirect the agent, providing a direct pathway for sensitive data to be exfiltrated. Yet, despite a growing number of real-world incidents, the confidentiality risks of such systems remain poorly understood. To address this gap, we provide a formalization of confidentiality in LLM-based agents. By abstracting sensitive data as a secret string, we evaluate ten agents across 20 tool scenarios and 14 attack strategies. We find that all agents are vulnerable to at least one attack, and existing defenses fail to provide reliable protection against these threats. Strikingly, we find that the tooling itself can amplify leakage risks.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 39 of its 91 references that are in the map. First posted on arXiv 10 Feb 2024.
Builds on
39 of 91 references.
Closest ancestors · 2
Same direction, newest first
- 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agents18 Jan
- 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep
Other references · 37
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
- 2020Extracting Training Data from Large Language Modelscited by 2,189
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
- 2021Ethical and social risks of harm from Language Modelscited by 966
- 2021Membership Inference Attacks From First Principlescited by 878
- 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
- 2016Stealing Machine Learning Models via Prediction APIscited by 619
25 more
- 2022Diffusion Models for Adversarial Purificationcited by 476
- 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
- 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 434
- 2023Detecting Language Model Attacks with Perplexitycited by 431
- 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
- 2018Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Modelscited by 362
- 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
- 2017MagNet: a Two-Pronged Defense against Adversarial Examplescited by 269
- 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 264
- 2017On Detecting Adversarial Perturbationscited by 232
- 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 230
- 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 226
- 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 216
- 2023On the Reliability of Watermarks for Large Language Modelscited by 189
- 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 183
- 2017On the (Statistical) Detection of Adversarial Examplescited by 180
- 2023Beyond Memorization: Violating Privacy Via Inference with Large Language Modelscited by 175
- 2014Towards Deep Neural Network Architectures Robust to Adversarial Examplescited by 146
- 2023Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMscited by 124
- 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 106
- 2022Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methodscited by 79
- 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competitioncited by 68
- 2017Enhancing Robustness of Machine Learning Systems via Data Transformationscited by 38
- 2023Beyond the Safeguards: Exploring the Security Risks of ChatGPTcited by 36
- 2023Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuningcited by 26
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation
- 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks
- 2025Automating Prompt Leakage Attacks on Large Language Models Using Agentic Approach
- 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows
- 2026When Agents Handle Secrets: A Survey of Confidential Computing for Agentic AI
- 2025Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Execution
- 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies
- 2025The Dark Side of LLMs: Agent-based Attack Vectors for System-level Compromise