mlsanityData to 7 Oct 2026

Fine-Tuning Large Language Models with User-Level Differential Privacy

arXiv 2407.07737 ↗First posted 10 Jul 2024cs.LG cs.CL cs.CRcited by 28

Not on the map. Why?

Abstract (arXiv)

We investigate practical and scalable algorithms for training large language models (LLMs) with user-level differential privacy (DP) in order to provably safeguard all the examples contributed by each user. We study two variants of DP-SGD with: (1) example-level sampling (ELS) and per-example gradient clipping, and (2) user-level sampling (ULS) and per-user gradient clipping. We derive a novel user-level DP accountant that allows us to compute provably tight privacy guarantees for ELS. Using this, we show that while ELS can outperform ULS in specific settings, ULS generally yields better results when each user has a diverse collection of examples. We validate our findings through experiments in synthetic mean estimation and LLM fine-tuning tasks under fixed compute budgets. We find that ULS is significantly better in settings where either (1) strong privacy guarantees are required, or (2) the compute budget is large. Notably, our focus on LLM-compatible training algorithms allows us to scale to models with hundreds of millions of parameters and datasets with hundreds of thousands of users.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 61 of its 79 references that are in the map. First posted on arXiv 10 Jul 2024.

Builds on

61 of 79 references.

References · 61

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2019Advances and Open Problems in Federated Learningcited by 3,021
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2016Deep Learning with Differential Privacycited by 2,682
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  10. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  11. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  12. 2022Quantifying Memorization Across Neural Language Modelscited by 755
38 more
  1. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  2. 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 451
  3. 2018Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learningcited by 242
  4. 2018Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergencescited by 237
  5. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  6. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 213
  7. 2021How Many Data Points is a Prompt Worth?cited by 197
  8. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 169
  9. 2018cpSGD: Communication-efficient and differentially-private distributed SGDcited by 157
  10. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  11. 2023Privacy Auditing with One (1) Training Runcited by 119
  12. 2019Gmail Smart Compose: Real-Time Assisted Writingcited by 97
  13. 2021Optimal Accounting of Differential Privacy via Characteristic Functioncited by 83
  14. 2021Learning with User-Level Privacycited by 72
  15. 2023Harnessing large-language models to generate private synthetic textcited by 65
  16. 2023Federated Learning of Gboard Language Models with Differential Privacycited by 64
  17. 2022Improved Differential Privacy for SGD via Optimal Private Linear Operators on Adaptive Streamscited by 59
  18. 2022TAN Without a Burn: Scaling Laws of DP-SGDcited by 51
  19. 2022Connect the Dots: Tighter Discrete Approximations of Privacy Loss Distributionscited by 50
  20. 2023(Amplified) Banded Matrix Factorization: A unified approach to private trainingcited by 49
  21. 2022Multi-Epoch Matrix Factorization Mechanisms for Private Machine Learningcited by 48
  22. 2020Tight Differential Privacy for Discrete-Valued Mechanisms and for the Subsampled Gaussian Mechanism Using FFTcited by 43
  23. 2020Learning discrete distributions: user vs item-level privacycited by 41
  24. 2019Differentially Private Meta-Learningcited by 40
  25. 2023User Inference Attacks on Large Language Modelscited by 36
  26. 2023One-shot Empirical Privacy Estimation for Federated Learningcited by 35
  27. 2023Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learningcited by 29
  28. 2021Tight and Robust Private Mean Estimation with Few Userscited by 26
  29. 2019The Audio Auditor: User-Level Membership Inference in Internet of Things Voice Servicescited by 25
  30. 2022Learning to Generate Image Embeddings with User-level Differential Privacycited by 23
  31. 2022User-Level Membership Inference Attack against Metric Embedding Learningcited by 22
  32. 2024How Private are DP-SGD Implementations?cited by 22
  33. 2024Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuningcited by 21
  34. 2023Unleashing the Power of Randomization in Auditing Differentially Private MLcited by 19
  35. 2023Privacy Amplification for Matrix Mechanismscited by 17
  36. 2023FACE-AUDITOR: Data Auditing in Facial Recognition Systemscited by 15
  37. 2020Optimal Private Median Estimation under Minimal Distributional Assumptionscited by 13
  38. 2023Federated Linear Contextual Bandits with User-level Differential Privacycited by 12

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 19; 2026: 5.

28 papers.

Sort:

By area:

  1. 2025Get Experience from Practice: LLM Agents with Record & Replaycited by 22
  2. 2025POPri: Private Federated Learning using Preference-Optimized Synthetic Datacited by 14
  3. 2024Evaluating Differentially Private Synthetic Data Generation in High-Stakes Domainscited by 12
  4. 2024A Hassle-free Algorithm for Private Learning in Practice: Don't Use Tree Aggregation, Use BLTscited by 9
  5. 2025On Design Principles for Private Adaptive Optimizerscited by 6
  6. 2025FedMentor: Domain-Aware Differential Privacy for Heterogeneous Federated LLMs in Mental Healthcited by 5
  7. 2025ContextLeak: Auditing Leakage in Private In-Context Learning Methodscited by 4
  8. 2025Empirical Privacy Variancecited by 3
  9. 2025DP-Adam-AC: Privacy-preserving Fine-Tuning of Localizable Language Models Using Adam Optimization with Adaptive Clippingcited by 3
  10. 2025Private-RAG: Answering Multiple Queries with LLMs while Keeping Your Data Privatecited by 2
  11. 2026JAX-Privacy: A library for differentially private machine learningcited by 2
  12. 2024Differential Privacy Regularization: Protecting Training Data Through Loss Function Regularizationcited by 1
16 more
  1. 2025Improved Algorithms for Differentially Private Language Model Alignmentcited by 1
  2. 2025SecP-Tuning: Efficient Privacy-Preserving Prompt Tuning for Large Language Models via MPCcited by 1
  3. 2025Enhancing Model Privacy in Federated Learning with Random Masking and Quantizationcited by 1
  4. 2025Forget What's Sensitive, Remember What Matters: Token-Level Differential Privacy in Memory Sculpting for Continual Learningcited by 1
  5. 2025Cocoon: A System Architecture for Differentially Private Training with Correlated Noisescited by 1
  6. 2026Toward Federated Large Language Models in Medicine: A Parameter-Efficient Framework for Privacy-Preserving, Multi-Institutional Adaptationcited by 1
  7. 2024Preserving Expert-Level Privacy in Offline Reinforcement Learningcited by 0
  8. 2025It's My Data Too: Private ML for Datasets with Multi-User Training Examplescited by 0
  9. 2025Black-Box Privacy Attacks on Shared Representations in Multitask Learningcited by 0
  10. 2025On the Inherent Privacy of Zeroth Order Projected Gradient Descentcited by 0
  11. 2025Auditable Early Stopping for Agentic Routing: Ledger-Verified Run-Wise Certificates under Local DPcited by 0
  12. 2025Secret-Protected Evolution for Differentially Private Synthetic Text Generationcited by 0
  13. 2025SA-ADP: Sensitivity-Aware Adaptive Differential Privacy for Large Language Modelscited by 0
  14. 2026PrivMedChat: End-to-End Differentially Private RLHF for Medical Dialogue Systemscited by 0
  15. 2026Active Inference as Context Acquisition for AI Agentscited by 0
  16. 2026Decoupling Knowledge and Privacy: Post-Task Self-Distillation Replay for LLM Continual Learningcited by 0