mlsanityData to 14 Oct 2026

Protecting Users From Themselves: Safeguarding Contextual Privacy in Interactions with Conversational Agents

arXiv 2502.18509 ↗First posted 22 Feb 2025cs.CR cs.AI cs.CLcited by 26

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Conversational agents are increasingly woven into individuals' personal lives, yet users often underestimate the privacy risks associated with them. The moment users share information with these agents-such as large language models (LLMs)-their private information becomes vulnerable to exposure. In this paper, we characterize the notion of contextual privacy for user interactions with LLM-based Conversational Agents (LCAs). It aims to minimize privacy risks by ensuring that users (sender) disclose only information that is both relevant and necessary for achieving their intended goals when interacting with LCAs (untrusted receivers). Through a formative design user study, we observe how even "privacy-conscious" users inadvertently reveal sensitive information through indirect disclosures. Based on insights from this study, we propose a locally deployable framework that operates between users and LCAs, identifying and reformulating out-of-context information in user prompts. Our evaluation using examples from ShareGPT shows that lightweight models can effectively implement this framework, achieving strong gains in contextual privacy while preserving the user's intended interaction goals. Notably, about 76% of participants in our human evaluation preferred the reformulated prompts over the original ones, validating the usability and effectiveness of contextual privacy in our proposed framework. We opensource the code at https://github.com/IBM/contextual-privacy-LLM.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 30 of its 70 references that are in the map. First posted on arXiv 22 Feb 2025.

Builds on

30 of 70 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023"It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents20 Sep

Other references · 29

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  2. 2020Extracting Training Data from Large Language Modelscited by 2,189
  3. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  4. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 434
  5. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  6. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  7. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  8. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 230
  9. 2021Counterfactual Memorization in Neural Language Modelscited by 156
  10. 2023Emergent and Predictable Memorization in Large Language Modelscited by 148
  11. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  12. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
17 more
  1. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 96
  2. 2023Machine Unlearning: Solutions and Challengescited by 93
  3. 2019Analyzing Information Leakage of Updates to Natural Language Modelscited by 90
  4. 2023DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineercited by 57
  5. 2024PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensemblescited by 57
  6. 2024Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Taskscited by 52
  7. 2023Reducing Privacy Risks in Online Self-Disclosures with Language Modelscited by 47
  8. 2024Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wildcited by 47
  9. 2024Operationalizing Contextual Integrity in Privacy-Conscious Assistantscited by 37
  10. 2022Differentially Private Decoding in Large Language Modelscited by 34
  11. 2024The Ethics of Interaction: Mitigating Security Threats in LLMscited by 33
  12. 2024Privacy Preserving Prompt Engineering: A Surveycited by 25
  13. 2023SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Modelscited by 15
  14. 2024DP-TabICL: In-Context Learning with Differentially Private Tabular Datacited by 11
  15. 2024Can LLMs get help from other LLMs without revealing private information?cited by 9
  16. 2024The Data Minimization Principle in Machine Learningcited by 7
  17. 2023Data Minimization at Inference Timecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 16.

26 papers.

Sort:

By area:

  1. 2025Towards Trustworthy GUI Agents: A Surveycited by 34
  2. 20251-2-3 Check: Enhancing Contextual Privacy in LLM via Multi-Agent Reasoningcited by 17
  3. 2025Privacy Reasoning in Ambiguous Contextscited by 11
  4. 2026AgentSCOPE: Evaluating Contextual Privacy Across Agentic Workflowscited by 10
  5. 2025Operationalizing Data Minimization for Privacy-Preserving LLM Promptingcited by 8
  6. 2026VoxPrivacy: A Benchmark for Evaluating Interactional Privacy of Speech Language Modelscited by 7
  7. 2025The Impact of a Chatbot's Ephemerality-Framing on Self-Disclosure Perceptionscited by 6
  8. 2026LCGuard: Latent Communication Guard for Safe KV Sharing in Multi-Agent Systemscited by 6
  9. 2025AI-Enabled Conversational Journaling for Advancing Parkinson's Disease Symptom Trackingcited by 4
  10. 2025Can Large Language Models Really Recognize Your Name?cited by 4
  11. 2025Personalizing Agent Privacy Decisions via Logical Entailmentcited by 4
  12. 2026CAPID: Context-Aware PII Detection for Question-Answering Systemscited by 3
14 more
  1. 2026Do LLMs Know What Is Private Internally? Probing and Steering Contextual Privacy Norms in Large Language Model Representationscited by 2
  2. 2026Local-First Intelligence via Proactive SLM Collaborating with Feedback-Providing LLMcited by 2
  3. 2026Minim: Privacy-Aware Minimal View for Agents via Trusted Local Sanitizationcited by 2
  4. 2025Black Box Absorption: LLMs Undermining Innovative Ideascited by 1
  5. 2025AudAgent: Automated Auditing of Privacy Policy Compliance in AI Agentscited by 1
  6. 2026Towards Privacy-Preserving Large Language Model: Text-free Inference Through Alignment and Adaptationcited by 1
  7. 2026What to Remember, What to Reveal: Privacy-Aware Memory for Conversational Agentscited by 1
  8. 2026Selection-Invariant Communication Compilers for Privacy-Aware Multi-Agent LLM Workflowscited by 1
  9. 2026Puda: Private User Dataset Agent for User-Sovereign and Privacy-Preserving Personalized AIcited by 0
  10. 2026A Privacy by Design Framework for Large Language Model-Based Applications for Childrencited by 0
  11. 2026Privacy Without Losing Place: A Paradigm for Private Retrieval in Spatial RAGscited by 0
  12. 2026GuardianAgent: Policy-Conditioned Risk-Adaptive Anonymization with Verified Adversarial Escalationcited by 0
  13. 2026Learning to Refer: Client-Resolved Generation for Privacy-Aware Language Modelscited by 0
  14. 2026PrivMeSA: Privacy-Aware Self-Evolving Multi-Agent System for Medicine via Local-Remote LLM Collaborationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MuPPET: A Benchmark for Contextual Privacy of LLM Assistants in Multi-Party Conversations
  2. 2026Say Something Else: Rethinking Contextual Privacy as Information Sufficiency
  3. 2024Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent
  4. 2025Beyond Jailbreaking: Auditing Contextual Privacy in LLM Agents
  5. 2025ConVerse: Benchmarking Contextual Safety in Agent-to-Agent Conversations
  6. 2024AirGapAgent: Protecting Privacy-Conscious Conversational Agents
  7. 2025MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation
  8. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action