mlsanityData to 7 Oct 2026

Differentially Private Language Models Benefit from Public Pre-training

arXiv 2009.05886 ↗First posted 13 Sep 2020cs.LG cs.CL cs.CRcited by 46

Not on the map. Why?

Abstract (arXiv)

Language modeling is a keystone task in natural language processing. When training a language model on sensitive information, differential privacy (DP) allows us to quantify the degree to which our private data is protected. However, training algorithms which enforce differential privacy often lead to degradation in model quality. We study the feasibility of learning a language model which is simultaneously high-quality and privacy preserving by tuning a public base model on a private corpus. We find that DP fine-tuning boosts the performance of language models in the private domain, making the training of such models possible.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 8; 2023: 13; 2024: 9; 2025: 10; 2026: 1.

46 papers.

Sort:

By area:

  1. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  2. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  3. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  4. 2023Locally Differentially Private Document Generation Using Zero Shot Promptingcited by 63
  5. 2021Selective Differential Privacy for Language Modelingcited by 61
  6. 2022THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryptioncited by 55
  7. 2022Exploring the Limits of Differentially Private Deep Learning with Group-wise Clippingcited by 52
  8. 2023Privacy-Preserving Parameter-Efficient Fine-Tuning for Large Language Model Servicescited by 43
  9. 2023Why Is Public Pretraining Necessary for Private Model Training?cited by 36
  10. 2022Submix: Practical Private Prediction for Large-Scale Language Modelscited by 31
  11. 2023Can Public Large Language Models Help Private Cross-device Federated Learning?cited by 30
  12. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
34 more
  1. 2023Split-and-Denoise: Protect large language model inference with local differential privacycited by 30
  2. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  3. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  4. 2024DP-MLM: Differentially Private Text Rewriting Using Masked Language Modelscited by 24
  5. 2023Selective Pre-training for Private Fine-tuningcited by 22
  6. 2022Sentence-level Privacy for Document Embeddingscited by 15
  7. 2024Fine-Tuning Language Models with Differential Privacy through Adaptive Noise Allocationcited by 14
  8. 2022Privacy-Preserving Domain Adaptation of Semantic Parserscited by 12
  9. 2022The Impact of Differential Privacy on Group Disparity Mitigationcited by 11
  10. 2021DPlis: Boosting Utility of Differentially Private Deep Learning via Randomized Smoothingcited by 10
  11. 2022Can Foundation Models Help Us Achieve Perfect Secrecy?cited by 9
  12. 2023Optimal Differentially Private Model Training with Public Datacited by 9
  13. 2024LLM-based Privacy Data Augmentation Guided by Knowledge Distillation with a Distribution Tutor for Medical Text Classificationcited by 8
  14. 2025On the Impact of Noise in Differentially Private Text Rewritingcited by 8
  15. 2023Efficiency 360: Efficient Vision Transformerscited by 6
  16. 2024Privacy Bias in Language Models: A Contextual Integrity-based Auditing Metriccited by 6
  17. 2025An Investigation of Memorization Risk in Healthcare Foundation Modelscited by 5
  18. 2025Dual-Priv Pruning : Efficient Differential Private Fine-Tuning in Multimodal Large Language Modelscited by 4
  19. 2023ConfusionPrompt: Practical Private Inference for Online Large Language Modelscited by 3
  20. 2024Differentially Private Learning Needs Better Model Initialization and Self-Distillationcited by 3
  21. 2021Learning Domain Specific Language Models for Automatic Speech Recognition through Machine Translationcited by 2
  22. 2023Context-Aware Differential Privacy for Language Modelingcited by 2
  23. 2023LatticeGen: A Cooperative Framework which Hides Generated Text in a Lattice for Privacy-Aware Generation on Cloudcited by 2
  24. 2025Towards Privacy-aware Mental Health AI Models: Advances, Challenges, and Opportunitiescited by 2
  25. 2025DP2Unlearning: An Efficient and Guaranteed Unlearning Framework for LLMscited by 2
  26. 2025PATCH: Mitigating PII Leakage in Language Models with Privacy-Aware Targeted Circuit PatcHingcited by 2
  27. 2023Differential Privacy, Linguistic Fairness, and Training Data Influence: Impossibility and Possibility Theorems for Multilingual Language Modelscited by 1
  28. 2024Privately Learning from Graphs with Applications in Fine-tuning Large Language Modelscited by 1
  29. 2025Empirical Calibration and Metric Differential Privacy in Language Modelscited by 1
  30. 2025NoEsis: Differentially Private Knowledge Transfer in Modular LLM Adaptationcited by 1
  31. 2025FlashDP: Private Training Large Language Models with Efficient DP-SGDcited by 1
  32. 2024IDT: Dual-Task Adversarial Attacks for Privacy Protectioncited by 0
  33. 2025Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)cited by 0
  34. 2026PrivBench: A Holistic and Modular Benchmarking Platform for Evaluating Text-to-Text Privatizationcited by 0