mlsanityData to 7 Oct 2026

Analyzing Leakage of Personally Identifiable Information in Language Models

arXiv 2302.00539 ↗First posted 1 Feb 2023cs.LGcited by 278

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

Language Models (LMs) have been shown to leak information about training data through sentence-level membership inference and reconstruction attacks. Understanding the risk of LMs leaking Personally Identifiable Information (PII) has received less attention, which can be attributed to the false assumption that dataset curation techniques such as scrubbing are sufficient to prevent PII leakage. Scrubbing techniques reduce but do not prevent the risk of PII leakage: in practice scrubbing is imperfect and must balance the trade-off between minimizing disclosure and preserving the utility of the dataset. On the other hand, it is unclear to which extent algorithmic defenses such as differential privacy, designed to guarantee sentence- or user-level privacy, prevent PII disclosure. In this work, we introduce rigorous game-based definitions for three types of PII leakage via black-box extraction, inference, and reconstruction attacks with only API access to an LM. We empirically evaluate the attacks against GPT-2 models fine-tuned with and without defenses in three domains: case law, health care, and e-mails. Our main contributions are (i) novel attacks that can extract up to 10× more PII sequences than existing attacks, (ii) showing that sentence-level differential privacy reduces the risk of PII disclosure but still leaks about 3% of PII sequences, and (iii) a subtle connection between record-level membership inference and PII reconstruction. Code to reproduce all experiments in the paper is available at https://github.com/microsoft/analysingpiileakage.

Key facts. Cited by 278 papers in the map (data through 7 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 48 of its 88 references that are in the map. First posted on arXiv 1 Feb 2023.

Builds on

48 of 88 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks8 Mar
  2. 2021Membership Inference Attacks From First Principles7 Dec
  3. 2021Membership Inference Attack Susceptibility of Clinical Language Models16 Apr

Other references · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2016Deep Learning with Differential Privacycited by 2,682
  10. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  11. 2020Extracting Training Data from Large Language Modelscited by 2,144
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
33 more
  1. 2018Hierarchical Neural Story Generationcited by 1,182
  2. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  3. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  4. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  5. 2016Neural Architectures for Named Entity Recognitioncited by 548
  6. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  7. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  8. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  9. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
  10. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  11. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  12. 2019Neural Legal Judgment Prediction in Englishcited by 208
  13. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  14. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
  15. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  16. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  17. 2019Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systemscited by 110
  18. 2019Gmail Smart Compose: Real-Time Assisted Writingcited by 97
  19. 2022The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymizationcited by 94
  20. 2018CGMH: Constrained Sentence Generation by Metropolis-Hastings Samplingcited by 93
  21. 2021Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?cited by 91
  22. 2019Analyzing Information Leakage of Updates to Natural Language Modelscited by 89
  23. 2022Do Language Models Plagiarize?cited by 72
  24. 2021Selective Differential Privacy for Language Modelingcited by 61
  25. 2022Memorization in NLP Fine-tuning Methodscited by 43
  26. 2022Defending against Reconstruction Attacks with R'enyi Differential Privacycited by 30
  27. 2022Text Revealer: Private Text Reconstruction via Model Inversion Attacks against Transformerscited by 28
  28. 2020Investigating Membership Inference Attacks under Data Dependenciescited by 27
  29. 2022Canary Extraction in Natural Language Understanding Modelscited by 20
  30. 2019Efficient Adaptation of Pretrained Transformers for Abstractive Summarizationcited by 13
  31. 2022Provably Confidential Language Modellingcited by 13
  32. 2021Training Data Leakage Analysis in Language Modelscited by 7
  33. 2022Combing for Credentials: Active Pattern Extraction from Smart Replycited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 47; 2024: 88; 2025: 84; 2026: 59.

278 papers.

Sort:

By area:

  1. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  2. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  3. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  4. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  5. 2023Challenges and Applications of Large Language Modelscited by 251
  6. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  7. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  8. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
  9. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  10. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  11. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  12. 2023Beyond Memorization: Violating Privacy Via Inference with Large Language Modelscited by 167
38 more
  1. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 156
  2. 2023ProPILE: Probing Privacy Leakage in Large Language Modelscited by 127
  3. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 121
  4. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  5. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 100
  6. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 96
  7. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  8. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 87
  9. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 85
  10. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  11. 2024Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systemscited by 76
  12. 2024Differentially Private Synthetic Data via Foundation Model APIs 2: Textcited by 74
  13. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 67
  14. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  15. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  16. 2024Building Guardrails for Large Language Modelscited by 61
  17. 2023DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineercited by 56
  18. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 54
  19. 2024PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensemblescited by 54
  20. 2024From Pixels to Prose: A Large Dataset of Dense Image Captionscited by 52
  21. 2023Did the Neurons Read your Book? Document-level Membership Inference for Large Language Modelscited by 51
  22. 2024NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surroundingcited by 49
  23. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  24. 2023Reducing Privacy Risks in Online Self-Disclosures with Language Modelscited by 47
  25. 2024Machine Unlearning for Image-to-Image Generative Modelscited by 47
  26. 2023Privacy-Preserving Parameter-Efficient Fine-Tuning for Large Language Model Servicescited by 43
  27. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 43
  28. 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Modelscited by 42
  29. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 42
  30. 2024How Does Quantization Affect Multilingual LLMs?cited by 41
  31. 2023ChatGPT Evaluation on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse Relationscited by 39
  32. 2024Voice Jailbreak Attacks Against GPT-4ocited by 39
  33. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 38
  34. 2023User Inference Attacks on Large Language Modelscited by 36
  35. 2023SoK: Memorization in General-Purpose Large Language Modelscited by 36
  36. 2024Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applicationscited by 36
  37. 2023The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Riskscited by 35
  38. 2023PrivLM-Bench: A Multi-level Privacy Evaluation Benchmark for Language Modelscited by 35

50 most cited of 278 All 278 (JSON)