mlsanityData to 14 Oct 2026

Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory

arXiv 2310.17884 ↗First posted 27 Oct 2023cs.AI cs.CL cs.CRcited by 230

Not on the map. Why?

Cited by 74 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The interactive use of large language models (LLMs) in AI assistants (at work, home, etc.) introduces a new set of inference-time privacy risks: LLMs are fed different types of information from multiple sources in their inputs and are expected to reason about what to share in their outputs, for what purpose and with whom, within a given context. In this work, we draw attention to the highly critical yet overlooked notion of contextual privacy by proposing ConfAIde, a benchmark designed to identify critical weaknesses in the privacy reasoning capabilities of instruction-tuned LLMs. Our experiments show that even the most capable models such as GPT-4 and ChatGPT reveal private information in contexts that humans would not, 39% and 57% of the time, respectively. This leakage persists even when we employ privacy-inducing prompts or chain-of-thought reasoning. Our work underscores the immediate need to explore novel inference-time privacy-preserving approaches, based on reasoning and theory of mind.

Key facts. Cited by 230 papers in the map (data through 14 Oct 2026). Builds on 33 of its 71 references that are in the map. First posted on arXiv 27 Oct 2023.

Builds on

33 of 71 references.

References · 33

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2016Deep Learning with Differential Privacycited by 2,728
  8. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  9. 2023Large Language Models as Optimizerscited by 864
  10. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  11. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  12. 2021Differentially Private Fine-tuning of Language Modelscited by 400
21 more
  1. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
  2. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 292
  3. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 255
  4. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  5. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 195
  6. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 171
  7. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 168
  8. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Modelscited by 163
  9. 2021Can Machines Learn Morality? The Delphi Experimentcited by 141
  10. 2022ProsocialDialog: A Prosocial Backbone for Conversational Agentscited by 109
  11. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 96
  12. 2023Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Trackercited by 95
  13. 2022When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgmentcited by 93
  14. 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 83
  15. 2018Evaluating Theory of Mind in Question Answeringcited by 64
  16. 2021Selective Differential Privacy for Language Modelingcited by 62
  17. 2023Can Language Models be Instructed to Protect Personal Information?cited by 46
  18. 2022ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situationscited by 36
  19. 2023Rethinking Machine Ethics -- Can LLMs Perform Moral Reasoning through the Lens of Moral Theories?cited by 36
  20. 2023COBRA Frames: Contextual Reasoning about Effects and Harms of Offensive Statementscited by 34
  21. 2023Are Chatbots Ready for Privacy-Sensitive Applications? An Investigation into Input Regurgitation and Prompt-Induced Sanitizationcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 49; 2025: 64; 2026: 114.

230 papers.

Sort:

By area:

  1. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  2. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 171
  3. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 150
  4. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 116
  5. 2025OS-Harm: A Benchmark for Measuring Safety of Computer Use Agentscited by 111
  6. 2024AirGapAgent: Protecting Privacy-Conscious Conversational Agentscited by 94
  7. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 93
  8. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  9. 2025AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agentscited by 82
  10. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 77
  11. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 77
  12. 2024Building Guardrails for Large Language Modelscited by 65
38 more
  1. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 59
  2. 2025Contextual Agent Security: A Policy for Every Purposecited by 59
  3. 2025The Obvious Invisible Threat: LLM-Powered GUI Agents' Vulnerability to Fine-Print Injectionscited by 59
  4. 2024PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensemblescited by 57
  5. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 53
  6. 2024CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreferencecited by 51
  7. 2024MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Controlcited by 51
  8. 2023Reducing Privacy Risks in Online Self-Disclosures with Language Modelscited by 47
  9. 2024Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wildcited by 47
  10. 2023Can LLMs Follow Simple Rules?cited by 46
  11. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 41
  12. 2024SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanationscited by 38
  13. 2024GoldCoin: Grounding Large Language Models in Privacy Laws via Contextual Integrity Theorycited by 37
  14. 2024Operationalizing Contextual Integrity in Privacy-Conscious Assistantscited by 37
  15. 2024CI-Bench: Benchmarking Contextual Integrity of AI Assistants on Synthetic Datacited by 37
  16. 2025PrivaCI-Bench: Evaluating Privacy with Contextual Integrity and Legal Compliancecited by 37
  17. 2024REEF: Representation Encoding Fingerprints for Large Language Modelscited by 36
  18. 2025Firewalls to Secure Dynamic LLM Agentic Networkscited by 36
  19. 2025Privacy in Action: Towards Realistic Privacy Mitigation and Evaluation for LLM-Powered Agentscited by 36
  20. 2024SPML: A DSL for Defending Language Models Against Prompt Attackscited by 34
  21. 2023How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilitiescited by 33
  22. 2024Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?cited by 30
  23. 2025MAGPIE: A benchmark for Multi-AGent contextual PrIvacy Evaluationcited by 29
  24. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 28
  25. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistencycited by 27
  26. 2025Protecting Users From Themselves: Safeguarding Contextual Privacy in Interactions with Conversational Agentscited by 26
  27. 2025SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attackscited by 23
  28. 2024Permissive Information-Flow Analysis for Large Language Modelscited by 22
  29. 2025Searching for Privacy Risks in LLM Agents via Simulationcited by 22
  30. 2024Granular Privacy Control for Geolocation with Vision Language Modelscited by 21
  31. 2024The Better Angels of Machine Personality: How Personality Relates to LLM Safetycited by 21
  32. 2025A Survey on Privacy Risks and Protection in Large Language Modelscited by 21
  33. 2025CASE-Bench: Context-Aware SafEty Benchmark for Large Language Modelscited by 20
  34. 2025The Sum Leaks More Than Its Parts: Compositional Privacy Risks and Mitigations in Multi-Agent Collaborationcited by 20
  35. 2024Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Modelscited by 18
  36. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 18
  37. 2025GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detectioncited by 17
  38. 20251-2-3 Check: Enhancing Contextual Privacy in LLM via Multi-Agent Reasoningcited by 17

50 most cited of 230 All 230 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SALT: Steering Activations towards Leakage-free Thinking in Chain of Thought
  2. 2024Preserving Privacy in Large Language Models: A Survey on Current Threats and Solutions
  3. 2025Contextual Integrity in LLMs via Reasoning and Reinforcement Learning
  4. 2026Loose Lips: Understanding Privacy Leakage in Agentic Dialogue
  5. 2025SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation
  6. 2024Privacy-Preserving Large Language Models: Mechanisms, Applications, and Future Directions
  7. 2025Beyond Jailbreaking: Auditing Contextual Privacy in LLM Agents
  8. 2025Assessing Visual Privacy Risks in Multimodal AI: A Novel Taxonomy-Grounded Evaluation of Vision-Language Models