mlsanityData to 7 Oct 2026

What Does it Mean for a Language Model to Preserve Privacy?

arXiv 2202.05520 ↗First posted 11 Feb 2022stat.ML cs.CL cs.LGcited by 194

Not on the map. Why?

Cited by 32 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Natural language reflects our private lives and identities, making its privacy concerns as broad as those of real life. Language models lack the ability to understand the context and sensitivity of text, and tend to memorize phrases present in their training sets. An adversary can exploit this tendency to extract training data. Depending on the nature of the content and the context in which this data was collected, this could violate expectations of privacy. Thus there is a growing interest in techniques for training language models that preserve privacy. In this paper, we discuss the mismatch between the narrow assumptions made by popular data protection techniques (data sanitization and differential privacy), and the broadness of natural language and of privacy as a social norm. We argue that existing protection methods cannot guarantee a generic and meaningful notion of privacy for language models. We conclude that language models should be trained on text data which was explicitly produced for public use.

Key facts. Cited by 194 papers in the map (data through 7 Oct 2026). Builds on 46 of its 109 references that are in the map. First posted on arXiv 11 Feb 2022.

Builds on

46 of 109 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2016Deep Learning with Differential Privacycited by 2,682
  9. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  10. 2020Extracting Training Data from Large Language Modelscited by 2,144
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  12. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
34 more
  1. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  2. 2021Ethical and social risks of harm from Language Modelscited by 957
  3. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  4. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  5. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  6. 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 475
  7. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  8. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  9. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  10. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  11. 2020Calibration of Pre-trained Transformerscited by 307
  12. 2016A Persona-Based Neural Conversation Modelcited by 221
  13. 2020Information Leakage in Embedding Modelscited by 199
  14. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 193
  15. 2019Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learningcited by 117
  16. 2021Large-Scale Differentially Private BERTcited by 117
  17. 2019Gmail Smart Compose: Real-Time Assisted Writingcited by 97
  18. 2021Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?cited by 91
  19. 2019Analyzing Information Leakage of Updates to Natural Language Modelscited by 89
  20. 2016De-identification of Patient Notes with Recurrent Neural Networkscited by 88
  21. 2017Ethical Challenges in Data-Driven Dialogue Systemscited by 86
  22. 2021Learning with User-Level Privacycited by 72
  23. 2021Differential Privacy Dynamics of Langevin Diffusion and Noisy Gradient Descentcited by 71
  24. 2020Training Production Language Models without Memorizing User Datacited by 68
  25. 2020ML Privacy Meter: Aiding Regulatory Compliance by Quantifying the Privacy Risks of Machine Learningcited by 66
  26. 2021Selective Differential Privacy for Language Modelingcited by 61
  27. 2020Handwritten Optical Character Recognition (OCR): A Comprehensive Systematic Literature Review (SLR)cited by 56
  28. 2020Like hiking? You probably enjoy nature: Persona-grounded Dialog with Commonsense Expansionscited by 49
  29. 2021Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Levelcited by 34
  30. 2021Membership Inference on Word Embedding and Beyondcited by 34
  31. 2017Learning Differentially Private Recurrent Language Modelscited by 32
  32. 2017Towards Measuring Membership Privacycited by 27
  33. 2015Neural CRF Parsingcited by 10
  34. 2018Auditing Data Provenance in Text-Generation Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 19; 2023: 38; 2024: 70; 2025: 49; 2026: 18.

194 papers.

Sort:

By area:

  1. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  2. 2023Extracting Training Data from Diffusion Modelscited by 723
  3. 2023A Comprehensive Overview of Large Language Modelscited by 639
  4. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  5. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  6. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  7. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 218
  8. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  9. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 168
  10. 2023Beyond Memorization: Violating Privacy Via Inference with Large Language Modelscited by 167
  11. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 161
  12. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 156
38 more
  1. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  2. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  3. 2022Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacycited by 97
  4. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  5. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 91
  6. 2024Guardrail Baselines for Unlearning in LLMscited by 90
  7. 2024AirGapAgent: Protecting Privacy-Conscious Conversational Agentscited by 90
  8. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbationscited by 88
  9. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 86
  10. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  11. 2023Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Reviewcited by 83
  12. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  13. 2025AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agentscited by 79
  14. 2024Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systemscited by 76
  15. 2023DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Modelscited by 75
  16. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  17. 2022Do Language Models Plagiarize?cited by 72
  18. 2022Differentially Private Language Models for Secure Data Sharingcited by 68
  19. 2022Mix and Match: Learning-free Controllable Text Generation using Energy Language Modelscited by 66
  20. 2024Clio: Privacy-Preserving Insights into Real-World AI Usecited by 66
  21. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  22. 2023"It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agentscited by 60
  23. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progresscited by 58
  24. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 54
  25. 2024PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensemblescited by 54
  26. 2024Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Datacited by 52
  27. 2024Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Datacited by 49
  28. 2023TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Taskscited by 48
  29. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  30. 2023Can Language Models be Instructed to Protect Personal Information?cited by 46
  31. 2022Memorization in NLP Fine-tuning Methodscited by 43
  32. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 43
  33. 2024To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Modelscited by 42
  34. 2023Beyond the Safeguards: Exploring the Security Risks of ChatGPTcited by 36
  35. 2023SoK: Memorization in General-Purpose Large Language Modelscited by 36
  36. 2024Operationalizing Contextual Integrity in Privacy-Conscious Assistantscited by 36
  37. 2022Just Fine-tune Twice: Selective Differential Privacy for Large Language Modelscited by 32
  38. 2024Participation in the age of foundation modelscited by 32

50 most cited of 194 All 194 (JSON)