mlsanityData to 7 Oct 2026

Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning

arXiv 2406.14322 ↗First posted 20 Jun 2024cs.CL cs.CR cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have emerged as powerful tools for tackling complex tasks across diverse domains, but they also raise privacy concerns when fine-tuned on sensitive data due to potential memorization. While differential privacy (DP) offers a promising solution by ensuring models are 'almost indistinguishable' with or without any particular privacy unit, current evaluations on LLMs mostly treat each example (text record) as the privacy unit. This leads to uneven user privacy guarantees when contributions per user vary. We therefore study user-level DP motivated by applications where it necessary to ensure uniform privacy protection across users. We present a systematic evaluation of user-level DP for LLM fine-tuning on natural language generation tasks. Focusing on two mechanisms for achieving user-level DP guarantees, Group Privacy and User-wise DP-SGD, we investigate design choices like data selection strategies and parameter tuning for the best privacy-utility tradeoff.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 42 of its 59 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

42 of 59 references.

References · 42

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2016Deep Learning with Differential Privacycited by 2,682
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  9. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  10. 2023PaLM 2 Technical Reportcited by 1,154
  11. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  12. 2023StarCoder: may the source be with you!cited by 790
30 more
  1. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  2. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  3. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  4. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  5. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
  6. 2018Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learningcited by 242
  7. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  8. 2021Numerical Composition of Differential Privacycited by 174
  9. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 169
  10. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  11. 2021Large-Scale Differentially Private BERTcited by 117
  12. 2023Foundation Models and Fair Usecited by 112
  13. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
  14. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 94
  15. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 87
  16. 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 80
  17. 2021Learning with User-Level Privacycited by 72
  18. 2022Connect the Dots: Tighter Discrete Approximations of Privacy Loss Distributionscited by 50
  19. 2023Privacy-Preserving In-Context Learning for Large Language Modelscited by 50
  20. 2020Tight Differential Privacy for Discrete-Valued Mechanisms and for the Subsampled Gaussian Mechanism Using FFTcited by 43
  21. 2023Privacy-Preserving Parameter-Efficient Fine-Tuning for Large Language Model Servicescited by 43
  22. 2020Learning discrete distributions: user vs item-level privacycited by 41
  23. 2023Privacy Implications of Retrieval-Based Language Modelscited by 31
  24. 2019Differentially Private SQL with Bounded User Contributioncited by 27
  25. 2022Learning to Generate Image Embeddings with User-level Differential Privacycited by 23
  26. 2024How Private are DP-SGD Implementations?cited by 22
  27. 2023User-Level Differential Privacy With Few Examples Per Usercited by 12
  28. 2023User-level Differentially Private Stochastic Convex Optimization: Efficient Algorithms with Optimal Ratescited by 12
  29. 2023On User-Level Private Convex Optimizationcited by 8
  30. 2024Tight Group-Level DP Guarantees for DP-SGD with Sampling via Mixture of Gaussians Mechanismscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 13; 2026: 4.

21 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2024Fine-Tuning Large Language Models with User-Level Differential Privacycited by 28
  3. 2025Towards User-level Private Reinforcement Learning with Human Feedbackcited by 13
  4. 2024Evaluating Differentially Private Synthetic Data Generation in High-Stakes Domainscited by 12
  5. 2024A Hassle-free Algorithm for Private Learning in Practice: Don't Use Tree Aggregation, Use BLTscited by 9
  6. 2025The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigationcited by 9
  7. 2025Optimizing Diversity and Quality through Base-Aligned Model Collaborationcited by 8
  8. 2025Gaussian DP for Reporting Differential Privacy Guarantees in Machine Learningcited by 6
  9. 2025ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Datacited by 6
  10. 2026Chain-of-Sanitized-Thoughts: Plugging PII Leakage in CoT of Large Reasoning Modelscited by 5
  11. 2025Empirical Privacy Variancecited by 3
  12. 2024Efficient and Private: Memorisation under differentially private parameter-efficient fine-tuning in language modelscited by 2
9 more
  1. 2025PrivacyXray: Detecting Privacy Breaches in LLMs through Semantic Consistency and Probability Certaintycited by 1
  2. 2025Correlating Cross-Iteration Noise for DP-SGD using Model Curvaturecited by 1
  3. 2025Cocoon: A System Architecture for Differentially Private Training with Correlated Noisescited by 1
  4. 2025It's My Data Too: Private ML for Datasets with Multi-User Training Examplescited by 0
  5. 2025Black-Box Privacy Attacks on Shared Representations in Multitask Learningcited by 0
  6. 2025Auditable Early Stopping for Agentic Routing: Ledger-Verified Run-Wise Certificates under Local DPcited by 0
  7. 2026SoK: Enhancing Cryptographic Collaborative Learning with Differential Privacycited by 0
  8. 2026Matrix Factorization for Practical Continual Mean Estimation Under User-Level Differential Privacycited by 0
  9. 2026Private PoEtry: Private In-Context Learning via Product of Expertscited by 0