mlsanityData to 14 Oct 2026

Agents That Know Too Much: A Data-Centric Survey of Privacy in LLM Agents

arXiv 2606.26627 ↗First posted 25 Jun 2026cs.CR cs.AIcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Large language model agents increasingly query databases, search document collections, call external APIs, remember past interactions, and act on a user's behalf. As they move from answering questions to operating over sensitive data, privacy becomes harder to enforce. An agent touches many data sources, runs multi-step workflows, keeps state across sessions, and acts with delegated permissions. Sensitive information can therefore leak not only through its final answer but through the queries it issues, the intermediate results it handles, the memory it writes, and the messages it exchanges with other agents. We survey the privacy of LLM agents from a data-centric view, organizing the field around the data an agent touches rather than by attack type, and we use data agent as shorthand for an LLM agent that works with data. Research on these risks is active but scattered across retrieval-augmented generation, text-to-SQL interfaces, agent memory, prompt injection, access control, and contextual privacy. This survey brings that work together: we taxonomize the data sources an agent touches, the privacy risks each source creates, and the governance mechanisms that address them; we map the benchmarks used to measure these risks and identify what is missing; and we set out the open problems. Two findings recur: among governance mechanisms only information-flow control covers both compositional and cross-session inference leakage, the two least-protected risks; and no benchmark drives an agent across its data surfaces under one privacy policy, the instrument the field most lacks. Our goal is a reference that situates the scattered literature and gives future work a common framing.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 75 of its 113 references that are in the map. First posted on arXiv 25 Jun 2026.

Builds on

75 of 113 references.

Closest ancestors · 25

Same direction, newest first

  1. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation9 Jun
  2. 2026AgentSecBench: Measuring Prompt Injection, Privacy Leakage, and Tool-Use Integrity in LLM Agents25 May
  3. 2026Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security17 May
  4. 2026AgentRaft: Automated Detection of Data Over-Exposure in LLM Agents8 Mar
  5. 2026Information-Theoretic Privacy Control for Sequential Multi-Agent LLM Systems13 Feb
  6. 2026Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Framework17 Jan
  7. 2025MAGPIE: A benchmark for Multi-AGent contextual PrIvacy Evaluation16 Oct
  8. 2025A Vision for Access Control in LLM-based Agent Systems13 Oct
  9. 2025Privacy in Action: Towards Realistic Privacy Mitigation and Evaluation for LLM-Powered Agents22 Sep
  10. 2025The Sum Leaks More Than Its Parts: Compositional Privacy Risks and Mitigations in Multi-Agent Collaboration16 Sep
  11. 20251-2-3 Check: Enhancing Contextual Privacy in LLM via Multi-Agent Reasoning11 Aug
  12. 2025Design Patterns for Securing LLM Agents against Prompt Injections10 Jun
13 more
  1. 2025Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Execution1 Jun
  2. 2025LlamaFirewall: An open source guardrail system for building secure AI agents6 May
  3. 2025Defeating Prompt Injections by Design24 Mar
  4. 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents24 Mar
  5. 2025AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents12 Mar
  6. 2025RTBAS: Defending LLM Agents Against Prompt Injection and Privacy Leakage13 Feb
  7. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents14 Nov
  8. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
  9. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents9 Oct
  10. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents3 Oct
  11. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies28 Jul
  12. 2024IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systems8 Mar
  13. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 50

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  4. 2020Extracting Training Data from Large Language Modelscited by 2,189
  5. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  6. 2019Machine Unlearningcited by 1,061
  7. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 533
  8. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 434
  9. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  10. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 285
  11. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 267
  12. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232
38 more
  1. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 230
  2. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 225
  3. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
  4. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 171
  5. 2024The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)cited by 159
  6. 2025Securing AI Agents with Information-Flow Controlcited by 121
  7. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 120
  8. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 96
  9. 2024System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspectivecited by 96
  10. 2024Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systemscited by 76
  11. 2023From Prompt Injections to SQL Injection Attacks: How Protected is Your LLM-Integrated Web Application?cited by 55
  12. 2024Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Datacited by 52
  13. 2024Privacy-Preserving Retrieval-Augmented Generation with Differential Privacycited by 39
  14. 2024GoldCoin: Grounding Large Language Models in Privacy Laws via Contextual Integrity Theorycited by 37
  15. 2024Operationalizing Contextual Integrity in Privacy-Conscious Assistantscited by 37
  16. 2024CI-Bench: Benchmarking Contextual Integrity of AI Assistants on Synthetic Datacited by 37
  17. 2025PrivaCI-Bench: Evaluating Privacy with Contextual Integrity and Legal Compliancecited by 37
  18. 2017Accuracy First: Selecting a Differential Privacy Level for Accuracy-Constrained ERMcited by 20
  19. 2024C-FedRAG: A Confidential Federated Retrieval-Augmented Generation Systemcited by 19
  20. 2022NLP-based Automated Compliance Checking of Data Processing Agreements against GDPRcited by 18
  21. 2025Position: Privacy Is Not Just Memorization!cited by 17
  22. 2013Blowfish Privacy: Tuning Privacy-Utility Trade-offs using Policiescited by 12
  23. 2018Discovering Smart Home Internet of Things Privacy Norms Using Contextual Integritycited by 11
  24. 2025Position: Contextual Integrity is Inadequately Applied to Language Modelscited by 11
  25. 2025Safeguarding Privacy of Retrieval Data against Membership Inference Attacks: Is This Query Too Close to Home?cited by 9
  26. 2025OrgAccess: A Benchmark for Role Based Access Control in Organization Scale LLMscited by 8
  27. 2025LLM-Based Data Science Agents: A Survey of Capabilities, Challenges, and Future Directionscited by 7
  28. 2024Unmasking Database Vulnerabilities: Zero-Knowledge Schema Inference Attacks in Text-to-SQL Systemscited by 6
  29. 2025Differentially Private Synthetic Text Generation for Retrieval-Augmented Generation (RAG)cited by 6
  30. 2025Role-Conditioned Refusals: Evaluating Access Control Reasoning in Large Language Modelscited by 5
  31. 2025PIR-RAG: A System for Private Information Retrieval in Retrieval-Augmented Generationcited by 4
  32. 2025SAFENLIDB: A Privacy-Preserving Safety Alignment Framework for LLM-based Natural Language Database Interfacescited by 4
  33. 2026An AI Agent Execution Environment to Safeguard User Datacited by 4
  34. 2019A Programming Framework for Differential Privacy with Accuracy Concentration Boundscited by 3
  35. 2025MaskSQL: Safeguarding Privacy for LLM-Based Text-to-SQL via Abstractioncited by 3
  36. 2025Data Guard: A Fine-grained Purpose-based Access Control System for Large Data Warehousescited by 2
  37. 2016Crowdsourced, Actionable and Verifiable Contextual Informational Normscited by 1
  38. 2024ProBE: Proportioning Privacy Budget for Complex Exploratory Decision Supportcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ADAM: A Systematic Data Extraction Attack on Agent Memory via Adaptive Querying
  2. 2026AIM: A Privacy-Aware Interoperable Memory Framework for Multi-Agent Multi-User LLM Systems
  3. 2025Exposing LLM User Privacy via Traffic Fingerprint Analysis: A Study of Privacy Risks in LLM Agent Interactions
  4. 2025MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation
  5. 2026AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems
  6. 2026NeuroFilter: Activation-Based Guardrails for Privacy-Conscious LLM Agents
  7. 2025Privacy-R1: Privacy-Aware Multi-LLM Agent Collaboration via Reinforcement Learning
  8. 2026OCELOT: Inference-Leakage Budgets for Privacy-Preserving LLM Agents