mlsanityData to 14 Oct 2026

Are Chatbots Ready for Privacy-Sensitive Applications? An Investigation into Input Regurgitation and Prompt-Induced Sanitization

arXiv 2305.15008 ↗First posted 24 May 2023cs.CL cs.AI cs.CYcited by 17

Not on the map. Why?

Abstract (arXiv)

LLM-powered chatbots are becoming widely adopted in applications such as healthcare, personal assistants, industry hiring decisions, etc. In many of these cases, chatbots are fed sensitive, personal information in their prompts, as samples for in-context learning, retrieved records from a database, or as part of the conversation. The information provided in the prompt could directly appear in the output, which might have privacy ramifications if there is sensitive information there. As such, in this paper, we aim to understand the input copying and regurgitation capabilities of these models during inference and how they can be directly instructed to limit this copying by complying with regulations such as HIPAA and GDPR, based on their internal knowledge of them. More specifically, we find that when ChatGPT is prompted to summarize cover letters of a 100 candidates, it would retain personally identifiable information (PII) verbatim in 57.4% of cases, and we find this retention to be non-uniform between different subgroups of people, based on attributes such as gender identity. We then probe ChatGPT's perception of privacy-related policies and privatization mechanisms by directly instructing it to provide compliant outputs and observe a significant omission of PII from output.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 22 of its 41 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

22 of 41 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  4. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
  5. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  6. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  7. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 285
  8. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 195
  9. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 175
  10. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  11. 2021Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?cited by 93
  12. 2016De-identification of Patient Notes with Recurrent Neural Networkscited by 88
10 more
  1. 2021Towards Improving Adversarial Training of NLP Modelscited by 79
  2. 2021Privacy Regularization: Joint Privacy-Utility Optimization in Language Modelscited by 30
  3. 2021Benchmarking Differential Privacy and Federated Learning for BERT Modelscited by 28
  4. 2022How to keep text private? A systematic review of deep learning methods for privacy-preserving natural language processingcited by 16
  5. 2023Ethical ChatGPT: Concerns, Challenges, and Commandmentscited by 14
  6. 2016Structured prediction models for RNN based sequence labeling in clinical textcited by 13
  7. 2023The moral authority of ChatGPTcited by 11
  8. 2022Combing for Credentials: Active Pattern Extraction from Smart Replycited by 7
  9. 2021NLP is Not enough -- Contextualization of User Input in Chatbotscited by 6
  10. 2020Is your chatbot GDPR compliant? Open issues in agent designcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 9; 2025: 1; 2026: 2.

17 papers.

Sort:

By area:

  1. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 230
  2. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 194
  3. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 96
  4. 2024Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wildcited by 47
  5. 2024Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMscited by 33
  6. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
  7. 20251-2-3 Check: Enhancing Contextual Privacy in LLM via Multi-Agent Reasoningcited by 17
  8. 2024DP-TabICL: In-Context Learning with Differentially Private Tabular Datacited by 11
  9. 2024Data-adaptive Differentially Private Prompt Synthesis for In-Context Learningcited by 10
  10. 2024FRACTURED-SORRY-Bench: Framework for Revealing Attacks in Conversational Turns Undermining Refusal Efficacy and Defenses over SORRY-Bench (Automated Multi-shot Jailbreaks)cited by 6
  11. 2024Estimating Privacy Leakage of Augmented Contextual Knowledge in Language Modelscited by 6
  12. 2024Best Practices for Large Language Models in Radiologycited by 5
5 more
  1. 2023From Military to Healthcare: Adopting and Expanding Ethical Principles for Generative Artificial Intelligencecited by 3
  2. 2024AERO: Entropy-Guided Framework for Private LLM Inferencecited by 3
  3. 2024Creation of AI-driven Smart Spaces for Enhanced Indoor Environments -- A Surveycited by 2
  4. 2026Got a Secret? LLM Agents Can't Keep It: Evaluating Privacy in Multi-Agent Systemscited by 1
  5. 2026Are Large Language Models Suitable for Graph Computation? Progress and Prospectscited by 0