mlsanityData to 7 Oct 2026

Neither Private Nor Fair: Impact of Data Imbalance on Utility and Fairness in Differential Privacy

arXiv 2009.06389 ↗First posted 10 Sep 2020cs.LG cs.AI cs.CRcited by 61

Not on the map. Why?

Abstract (arXiv)

Deployment of deep learning in different fields and industries is growing day by day due to its performance, which relies on the availability of data and compute. Data is often crowd-sourced and contains sensitive information about its contributors, which leaks into models that are trained on it. To achieve rigorous privacy guarantees, differentially private training mechanisms are used. However, it has recently been shown that differential privacy can exacerbate existing biases in the data and have disparate impacts on the accuracy of different subgroups of data. In this paper, we aim to study these effects within differentially private deep learning. Specifically, we aim to study how different levels of imbalance in the data affect the accuracy and the fairness of the decisions made by the model, given different levels of privacy. We demonstrate that even small imbalances and loose privacy guarantees can cause disparate impacts.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 9; 2023: 15; 2024: 14; 2025: 12; 2026: 4.

61 papers.

Sort:

  1. 2021Robin Hood and Matthew Effects: Differential Privacy Has Disparate Impact on Synthetic Datacited by 48
  2. 2022Differential Privacy and Fairness in Decisions and Learning Tasks: A Surveycited by 47
  3. 2021DP-SGD vs PATE: Which Has Less Disparate Impact on Model Accuracy?cited by 32
  4. 2022Disparate Impact in Differential Privacy from Gradient Misalignmentcited by 31
  5. 2021Interpretable Data-Based Explanations for Fairness Debuggingcited by 30
  6. 2021Privacy Regularization: Joint Privacy-Utility Optimization in Language Modelscited by 29
  7. 2023Privacy and Fairness in Federated Learning: on the Perspective of Trade-offcited by 26
  8. 2022Differential Privacy has Bounded Impact on Fairness in Classificationcited by 18
  9. 2023Private, fair and accurate: Training large-scale, privacy-preserving AI models in medical imagingcited by 17
  10. 2023Unlocking Accuracy and Fairness in Differentially Private Image Classificationcited by 16
  11. 2022Demographic-Reliant Algorithmic Fairness: Characterizing the Risks of Demographic Data Collection in the Pursuit of Fairnesscited by 13
  12. 2021Style Pooling: Automatic Text Style Obfuscation for Improved Classification Fairnesscited by 12
38 more
  1. 2023Recent Advances of Differential Privacy in Centralized Deep Learning: A Systematic Surveycited by 12
  2. 2024The Elusive Pursuit of Reproducing PATE-GAN: Benchmarking, Auditing, Debuggingcited by 12
  3. 2021A Fairness Analysis on Private Aggregation of Teacher Ensemblescited by 11
  4. 2022The Impact of Differential Privacy on Group Disparity Mitigationcited by 11
  5. 2023Learning with Impartiality to Walk on the Pareto Frontier of Fairness, Privacy, and Utilitycited by 10
  6. 2024Incentives in Private Collaborative Machine Learningcited by 10
  7. 2022SoK: Differential Privacy on Graph-Structured Datacited by 9
  8. 2022VisCUIT: Visual Auditor for Bias in CNN Image Classifiercited by 8
  9. 2021DP-SGD vs PATE: Which Has Less Disparate Impact on GANs?cited by 6
  10. 2023Preserving privacy in domain transfer of medical AI models comes at no performance costs: The integral role of differential privacycited by 6
  11. 2024A Systematic and Formal Study of the Impact of Local Differential Privacy on Fairness: Preliminary Resultscited by 6
  12. 2025Trustworthy Machine Learning via Memorization and the Granular Long-Tail: A Survey on Interactions, Tradeoffs, and Beyondcited by 6
  13. 2023Protect Your Score: Contact Tracing With Differential Privacy Guaranteescited by 5
  14. 2023On the Fairness Impacts of Private Ensembles Modelscited by 4
  15. 2023Trade-Offs Between Fairness and Privacy in Language Modelingcited by 4
  16. 2023Toward the Tradeoffs between Privacy, Fairness and Utility in Federated Learningcited by 4
  17. 2023On the Impact of Multi-dimensional Local Differential Privacy on Fairnesscited by 4
  18. 2023SoK: Taming the Triangle -- On the Interplays between Fairness, Interpretability and Privacy in Machine Learningcited by 4
  19. 2024Differentially Private Clustered Federated Learningcited by 4
  20. 2024Differential privacy enables fair and accurate AI-based analysis of speech disorders while protecting patient datacited by 4
  21. 2022Exploring the Unfairness of DP-SGD Across Settingscited by 3
  22. 2023Incentivising the federation: gradient-based metrics for data selection and valuation in private decentralised trainingcited by 3
  23. 2024Regulation Games for Trustworthy Machine Learningcited by 3
  24. 2024Differentially Private Fair Binary Classificationscited by 3
  25. 2024PFGuard: A Generative Framework with Privacy and Fairness Safeguardscited by 3
  26. 2025SoK: What Makes Private Learning Unfair?cited by 3
  27. 2022How Much User Context Do We Need? Privacy by Design in Mental Health NLP Applicationcited by 2
  28. 2024Disparate Impact on Group Accuracy of Linearization for Private Inferencecited by 2
  29. 2024Convex Approximation of Two-Layer ReLU Networks for Hidden State Differential Privacycited by 2
  30. 2025Learning with Differentially Private (Sliced) Wasserstein Gradientscited by 2
  31. 2024Privacy for Fairness: Information Obfuscation for Fair Representation Learning with Local Differential Privacycited by 1
  32. 2025On the Origins of Sampling Bias: Implications on Fairness Measurement and Mitigationcited by 1
  33. 2025SoftAdaClip: A Smooth Clipping Strategy for Fair and Private Model Trainingcited by 1
  34. 2025Private and Fair Machine Learning: Revisiting the Disparate Impact of Differentially Private SGDcited by 1
  35. 2026Where to Intervene? Benchmarking Fairness-Aware Learning on Differentially Private Synthetic Tabular Datacited by 1
  36. 2023Shedding light on underrepresentation and Sampling Bias in machine learningcited by 0
  37. 2023DP-SGD-Global-Adapt-V2-S: Triad Improvements of Privacy, Accuracy and Fairness via Step Decay Noise Multiplier and Step Decay Upper Clipping Thresholdcited by 0
  38. 2024On the Impact of Output Perturbation on Fairness in Binary Linear Classificationcited by 0

50 most cited of 61 All 61 (JSON)