# Selective Differential Privacy for Language Modeling

arXiv: https://arxiv.org/abs/2108.12944 · HTML: https://mlsanity.com/paper/2108.12944/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 30 Aug 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Renyi Differential Privacy](https://mlsanity.com/paper/1702.07476/) (2017)
- [Scalable Private Learning with PATE](https://mlsanity.com/paper/1802.08908/) (2018)
- [Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergences](https://mlsanity.com/paper/1807.01647/) (2018)
- [Differentially Private Learning with Adaptive Clipping](https://mlsanity.com/paper/1905.03871/) (2019)
- [Tempered Sigmoid Activations for Deep Learning with Differential Privacy](https://mlsanity.com/paper/2007.14191/) (2020)
- [Differential Privacy for Text Analytics via Natural Text Sanitization](https://mlsanity.com/paper/2106.01221/) (2021)
- [Training Production Language Models without Memorizing User Data](https://mlsanity.com/paper/2009.10031/) (2020)
- [Generalised Differential Privacy for Text Document Processing](https://mlsanity.com/paper/1811.10256/) (2018)
- [Differentially Private Language Models Benefit from Public Pre-training](https://mlsanity.com/paper/2009.05886/) (2020)
- [A Differentially Private Text Perturbation Method Using a Regularized Mahalanobis Metric](https://mlsanity.com/paper/2010.11947/) (2020)
- [Zero-Shot Dialog Generation with Cross-Domain Latent Actions](https://mlsanity.com/paper/1805.04803/) (2018)
- [Towards Automatic Generation of Shareable Synthetic Clinical Notes Using Neural Language Models](https://mlsanity.com/paper/1905.07002/) (2019)
- [DP-LSSGD: A Stochastic Optimization Method to Lift the Utility in Privacy-Preserving ERM](https://mlsanity.com/paper/1906.12056/) (2019)
- [A Privacy-Preserving Approach to Extraction of Personal Information through Automatic Annotation and Federated Learning](https://mlsanity.com/paper/2105.09198/) (2021)
- [One-sided Differential Privacy](https://mlsanity.com/paper/1712.05888/) (2017)
- [DPlis: Boosting Utility of Differentially Private Deep Learning via Randomized Smoothing](https://mlsanity.com/paper/2103.01496/) (2021)
- [Privacy Guarantees for De-identifying Text Transformations](https://mlsanity.com/paper/2008.03101/) (2020)
- [Differentially Private Deep Learning with Direct Feedback Alignment](https://mlsanity.com/paper/2010.03701/) (2020)

## Built on by (most cited first, 30 of 62)

- [Analyzing Leakage of Personally Identifiable Information in Language Models](https://mlsanity.com/paper/2302.00539/) (2023)
- [Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory](https://mlsanity.com/paper/2310.17884/) (2023)
- [Are Large Pre-Trained Language Models Leaking Your Personal Information?](https://mlsanity.com/paper/2205.12628/) (2022)
- [What Does it Mean for a Language Model to Preserve Privacy?](https://mlsanity.com/paper/2202.05520/) (2022)
- [Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey](https://mlsanity.com/paper/2210.07700/) (2022)
- [Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems](https://mlsanity.com/paper/2401.05778/) (2024)
- [Privacy-Preserving Parameter-Efficient Fine-Tuning for Large Language Model Services](https://mlsanity.com/paper/2305.06212/) (2023)
- [Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning](https://mlsanity.com/paper/2312.17493/) (2023)
- [SoK: Memorization in General-Purpose Large Language Models](https://mlsanity.com/paper/2310.18362/) (2023)
- [Large Language Models for UAVs: Current State and Pathways to the Future](https://mlsanity.com/paper/2405.01745/) (2024)
- [Just Fine-tune Twice: Selective Differential Privacy for Large Language Models](https://mlsanity.com/paper/2204.07667/) (2022)
- [Privacy Implications of Retrieval-Based Language Models](https://mlsanity.com/paper/2305.14888/) (2023)
- [Submix: Practical Private Prediction for Large-Scale Language Models](https://mlsanity.com/paper/2201.00971/) (2022)
- [Undesirable Memorization in Large Language Models: A Survey](https://mlsanity.com/paper/2410.02650/) (2024)
- [On the Impossible Safety of Large AI Models](https://mlsanity.com/paper/2209.15259/) (2022)
- [PrivacyRestore: Privacy-Preserving Inference in Large Language Models via Privacy Removal and Restoration](https://mlsanity.com/paper/2406.01394/) (2024)
- [PQLM -- Multilingual Decentralized Portable Quantum Language Model for Privacy Protection](https://mlsanity.com/paper/2210.03221/) (2022)
- [Can Copyright be Reduced to Privacy?](https://mlsanity.com/paper/2305.14822/) (2023)
- [Grounding Foundation Models through Federated Transfer Learning: A General Framework](https://mlsanity.com/paper/2311.17431/) (2023)
- [Differentially Private Natural Language Models: Recent Advances and Future Directions](https://mlsanity.com/paper/2301.09112/) (2023)
- [PrivacyMind: Large Language Models Can Be Contextual Privacy Protection Learners](https://mlsanity.com/paper/2310.02469/) (2023)
- [Provably Confidential Language Modelling](https://mlsanity.com/paper/2205.01863/) (2022)
- [Privacy-Preserving Domain Adaptation of Semantic Parsers](https://mlsanity.com/paper/2212.10520/) (2022)
- [Evaluating Differentially Private Synthetic Data Generation in High-Stakes Domains](https://mlsanity.com/paper/2410.08327/) (2024)
- [1-Diffractor: Efficient and Utility-Preserving Text Obfuscation Leveraging Word-Level Metric Differential Privacy](https://mlsanity.com/paper/2405.01678/) (2024)
- [PRISM: Privacy-Aware Routing for Adaptive Cloud-Edge LLM Inference via Semantic Sketch Collaboration](https://mlsanity.com/paper/2511.22788/) (2025)
- [Moderating Model Marketplaces: Platform Governance Puzzles for AI Intermediaries](https://mlsanity.com/paper/2311.12573/) (2023)
- [Just Rewrite It Again: A Post-Processing Method for Enhanced Semantic Similarity and Privacy Preservation of Differentially Private Rewritten Text](https://mlsanity.com/paper/2405.19831/) (2024)
- [Noise-Aware Algorithm for Heterogeneous Differentially Private Federated Learning](https://mlsanity.com/paper/2406.03519/) (2024)
- [LLM-based Privacy Data Augmentation Guided by Knowledge Distillation with a Distribution Tutor for Medical Text Classification](https://mlsanity.com/paper/2402.16515/) (2024)

## Abstract (arXiv)

With the increasing applications of language models, it has become crucial to protect these models from leaking private information. Previous work has attempted to tackle this challenge by training RNN-based language models with differential privacy guarantees. However, applying classical differential privacy to language models leads to poor model performance as the underlying privacy notion is over-pessimistic and provides undifferentiated protection for all tokens in the data. Given that the private information in natural language is sparse (for example, the bulk of an email might not carry personally identifiable information), we propose a new privacy notion, selective differential privacy, to provide rigorous privacy guarantees on the sensitive portion of the data to improve model utility. To realize such a new notion, we develop a corresponding privacy mechanism, Selective-DPSGD, for RNN-based language models. Besides language modeling, we also apply the method to a more concrete application--dialog systems. Experiments on both language modeling and dialog system building show that the proposed privacy-preserving mechanism achieves better utilities while remaining safe under various privacy attacks compared to the baselines. The data and code are released at https://github.com/wyshi/lm_privacy to facilitate future research .