mlsanityData to 14 Oct 2026

Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent

arXiv 2411.01344 ↗First posted 2 Nov 2024cs.HC cs.AI cs.CRcited by 14

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Language model (LM) agents that act on users' behalf for personal tasks (e.g., replying emails) can boost productivity, but are also susceptible to unintended privacy leakage risks. We present the first study on people's capacity to oversee the privacy implications of the LM agents. By conducting a task-based survey (N=300), we investigate how people react to and assess the response generated by LM agents for asynchronous interpersonal communication tasks, compared with a response they wrote. We found that people may favor the agent response with more privacy leakage over the response they drafted or consider both good, leading to an increased harmful disclosure from 15.7% to 55.0%. We further identified six privacy behavior patterns reflecting varying concerns, trust levels, and privacy preferences underlying people's oversight of LM agents' actions. Our findings shed light on designing agentic systems that enable privacy-preserving interactions and achieve bidirectional alignment on privacy preferences to help users calibrate trust.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 21 of its 74 references that are in the map. First posted on arXiv 2 Nov 2024.

Builds on

21 of 74 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep
  2. 2023"It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents20 Sep

Other references · 19

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  3. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  4. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  5. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  6. 2020Artificial Intelligence, Values and Alignmentcited by 448
  7. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agentscited by 333
  8. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  9. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 230
  10. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 171
  11. 2024The Ethics of Advanced AI Assistantscited by 107
  12. 2024On scalable oversight with weak LLMs judging strong LLMscited by 99
7 more
  1. 2023Eliciting Human Preferences with Language Modelscited by 92
  2. 2021Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviorscited by 50
  3. 2024Operationalizing Contextual Integrity in Privacy-Conscious Assistantscited by 37
  4. 2024On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectivescited by 28
  5. 2024Privacy Checklist: Privacy Violation Detection Grounding on Contextual Integrity Theorycited by 18
  6. 2017The Feasibility of Dynamically Granted Permissions: Aligning Mobile Privacy with User Preferencescited by 9
  7. 2024AI Delegates with a Dual Focus: Ensuring Privacy and Strategic Self-Disclosurecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 3.

14 papers.

Sort:

By area:

  1. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 96
  2. 2025UXAgent: An LLM Agent-Based Usability Testing Framework for Web Designcited by 40
  3. 2025Position: Privacy Is Not Just Memorization!cited by 17
  4. 2025Toward a Human-Centered Evaluation Framework for Trustworthy LLM-Powered GUI Agentscited by 11
  5. 2025UXAgent: A System for Simulating Usability Testing of Web Design with LLM Agentscited by 10
  6. 2025LLM-as-a-Judge for Privacy Evaluation? Exploring the Alignment of Human and LLM Perceptions of Privacy in Textual Datacited by 10
  7. 2025Beyond Data Privacy: New Privacy Risks for Large Language Modelscited by 6
  8. 2026Comparing Human Oversight Strategies for Computer-Use Agentscited by 6
  9. 2025DPRF: A Generalizable Dynamic Persona Refinement Framework for Optimizing Behavior Alignment Between Personalized LLM Role-Playing Agents and Humanscited by 5
  10. 2025It's My Agent, Not Me: Personal Privacy Boundaries in AI-Delegated Information Sharingcited by 3
  11. 2025Autonomy Reshapes How Personalization Affects Privacy Concerns and Trust in LLM Agentscited by 1
  12. 2025User Perceptions vs. Proxy LLM Judges: Privacy and Helpfulness in LLM Responses to Privacy-Sensitive Scenarioscited by 1
2 more
  1. 2026ToolMinimize: Auditing and Rewriting LLM Agent Tool Calls to Minimize Privacy Exposurecited by 1
  2. 2026Behavioral Transfer in AI Agents: Evidence and Privacy Implicationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Protecting Users From Themselves: Safeguarding Contextual Privacy in Interactions with Conversational Agents
  2. 2026PrivacyAlign: Contextual Privacy Alignment for LLM Agents
  3. 2026Loose Lips: Understanding Privacy Leakage in Agentic Dialogue
  4. 2026NeuroFilter: Activation-Based Guardrails for Privacy-Conscious LLM Agents
  5. 2026PrivacyReasoner: Can LLM Emulate a Human-like Privacy Mind?
  6. 2025Measuring Physical-World Privacy Awareness of Large Language Models: An Evaluation Benchmark
  7. 2025Beyond Jailbreaking: Auditing Contextual Privacy in LLM Agents
  8. 2026Got a Secret? LLM Agents Can't Keep It: Evaluating Privacy in Multi-Agent Systems