# Does Differential Privacy Impact Bias in Pretrained NLP Models?

arXiv: https://arxiv.org/abs/2410.18749 · HTML: https://mlsanity.com/paper/2410.18749/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 24 Oct 2024.

## Builds on (most cited first, 18 of 18)

- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification](https://mlsanity.com/paper/1903.04561/) (2019)
- [Differentially Private Fine-tuning of Language Models](https://mlsanity.com/paper/2110.06500/) (2021)
- [Opacus: User-Friendly Differential Privacy Library in PyTorch](https://mlsanity.com/paper/2109.12298/) (2021)
- [Differential Privacy Has Disparate Impact on Model Accuracy](https://mlsanity.com/paper/1905.12101/) (2019)
- [Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems](https://mlsanity.com/paper/1805.04508/) (2018)
- [Racial Bias in Hate Speech and Abusive Language Detection Datasets](https://mlsanity.com/paper/1905.12516/) (2019)
- [Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.03310/) (2019)
- [Large-Scale Differentially Private BERT](https://mlsanity.com/paper/2108.01624/) (2021)
- [A General Approach to Adding Differential Privacy to Iterative Training Procedures](https://mlsanity.com/paper/1812.06210/) (2018)
- [Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics](https://mlsanity.com/paper/2106.14574/) (2021)
- [Measuring a hate speech spectrum with faceted Rasch item response theory and perspective-aware, explainable-by-design deep learning](https://mlsanity.com/paper/2009.10277/) (2020)
- [Differentially Private Representation for NLP: Formal Guarantee and An Empirical Study on Privacy and Fairness](https://mlsanity.com/paper/2010.01285/) (2020)
- [DP-SGD vs PATE: Which Has Less Disparate Impact on Model Accuracy?](https://mlsanity.com/paper/2106.12576/) (2021)
- [Does Robustness Improve Fairness? Approaching Fairness with Word Substitution Robustness Methods for Text Classification](https://mlsanity.com/paper/2106.10826/) (2021)

## Built on by (most cited first, 2 of 2)

- [Differentially-private text generation degrades output language quality](https://mlsanity.com/paper/2509.11176/) (2025)
- [How Does Differential Privacy Affect Social Bias in LLMs? A Systematic Evaluation](https://mlsanity.com/paper/2605.11195/) (2026)

## Abstract (arXiv)

Differential privacy (DP) is applied when fine-tuning pre-trained large language models (LLMs) to limit leakage of training examples. While most DP research has focused on improving a model's privacy-utility tradeoff, some find that DP can be unfair to or biased against underrepresented groups. In this work, we show the impact of DP on bias in LLMs through empirical analysis. Differentially private training can increase the model bias against protected groups w.r.t AUC-based bias metrics. DP makes it more difficult for the model to differentiate between the positive and negative examples from the protected groups and other groups in the rest of the population. Our results also show that the impact of DP on bias is not only affected by the privacy protection level but also the underlying distribution of the dataset.