mlsanityData to 7 Oct 2026

Differential Privacy Dynamics of Langevin Diffusion and Noisy Gradient Descent

arXiv 2102.05855 ↗First posted 11 Feb 2021stat.ML cs.CR cs.LGcited by 71

Not on the map. Why?

Abstract (arXiv)

What is the information leakage of an iterative randomized learning algorithm about its training data, when the internal state of the algorithm is private? How much is the contribution of each specific training epoch to the information leakage through the released model? We study this problem for noisy gradient descent algorithms, and model the dynamics of R'enyi differential privacy loss throughout the training process. Our analysis traces a provably tight bound on the R'enyi divergence between the pair of probability distributions over parameters of models trained on neighboring datasets. We prove that the privacy loss converges exponentially fast, for smooth and strongly convex loss functions, which is a significant improvement over composition theorems (which over-estimate the privacy loss by upper-bounding its total value over all intermediate gradient computations). For Lipschitz, smooth, and strongly convex loss functions, we prove optimal utility with a small gradient complexity for noisy gradient descent algorithms.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Builds on 20 of its 40 references that are in the map. First posted on arXiv 11 Feb 2021.

Builds on

20 of 40 references.

References · 20

  1. 2016Deep Learning with Differential Privacycited by 2,682
  2. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  3. 2020Extracting Training Data from Large Language Modelscited by 2,144
  4. 2017Renyi Differential Privacycited by 643
  5. 2018Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergencescited by 237
  6. 2018Subsampled R'enyi Differential Privacy and Analytical Moments Accountantcited by 234
  7. 2019Rapid Convergence of the Unadjusted Langevin Algorithm: Isoperimetry Sufficescited by 224
  8. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 213
  9. 2014Theoretical guarantees for approximate sampling from smooth and log-concave densitiescited by 206
  10. 2015Non-asymptotic convergence analysis for the Unadjusted Langevin Algorithmcited by 195
  11. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 193
  12. 2020Private Stochastic Convex Optimization: Optimal Rates in Linear Timecited by 152
8 more
  1. 2018Differentially Private Empirical Risk Minimization Revisited: Faster and More Generalcited by 127
  2. 2018Privacy Amplification by Iterationcited by 97
  3. 2015Privacy for Free: Posterior Sampling and Stochastic Gradient Monte Carlocited by 61
  4. 2017Efficient Private ERM for Smooth Objectivescited by 61
  5. 2019On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learningcited by 57
  6. 2020Faster Differentially Private Samplers via R'enyi Divergence Analysis of Discretized Langevin MCMCcited by 27
  7. 2019Privacy Amplification by Mixing and Diffusion Mechanismscited by 21
  8. 2020Privacy Amplification of Iterative Algorithms via Contraction Coefficientscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 13; 2023: 12; 2024: 23; 2025: 13; 2026: 9.

71 papers.

Sort:

By area:

  1. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
  2. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  3. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  4. 2022Privacy of Noisy Stochastic Gradient Descent: More Iterations without More Privacy Losscited by 60
  5. 2022Forget Unlearning: Towards True Data-Deletion in Machine Learningcited by 58
  6. 2023(Amplified) Banded Matrix Factorization: A unified approach to private trainingcited by 49
  7. 2024Langevin Unlearning: A New Perspective of Noisy Gradient Descent for Machine Unlearningcited by 48
  8. 2022Differentially Private Learning Needs Hidden State (Or Much Faster Convergence)cited by 45
  9. 2023Faster high-accuracy log-concave sampling via algorithmic warm startscited by 44
  10. 2022Bayesian Estimation of Differential Privacycited by 31
  11. 2022Bring Your Own Algorithm for Optimal Differentially Private Stochastic Minimax Optimizationcited by 22
  12. 2024How Private are DP-SGD Implementations?cited by 22
38 more
  1. 2022Resolving the Mixing Time of the Langevin Algorithm to its Stationary Distribution for Log-Concave Samplingcited by 21
  2. 2024Privacy Backdoors: Stealing Data with Corrupted Pretrained Modelscited by 20
  3. 2024Nearly Tight Black-Box Auditing of Differentially Private Machine Learningcited by 19
  4. 2022Differential Privacy Guarantees for Stochastic Gradient Langevin Dynamicscited by 18
  5. 2022Self-Consistency of the Fokker-Planck Equationcited by 18
  6. 2023Exploring the Benefits of Visual Prompting in Differential Privacycited by 17
  7. 2024Certified Machine Unlearning via Noisy Stochastic Gradient Descentcited by 13
  8. 2023Private (Stochastic) Non-Convex Optimization Revisited: Second-Order Stationary Points and Excess Riskscited by 12
  9. 2023DIFF2: Differential Private Optimization via Gradient Differences for Nonconvex Distributed Learningcited by 10
  10. 2023Privacy Loss of Noisy Stochastic Gradient Descent Might Converge Even for Non-Convex Lossescited by 10
  11. 2024Shifted Interpolation for Differential Privacycited by 10
  12. 2024It's Our Loss: No Privacy Amplification for Hidden State DP-SGD With Non-Convex Losscited by 10
  13. 2024The Last Iterate Advantage: Empirical Auditing and Principled Heuristic Analysis of Differentially Private SGDcited by 10
  14. 2022On the Statistical Complexity of Estimation and Testing under Privacy Constraintscited by 6
  15. 2024Convergent Privacy Loss of Noisy-SGD without Convexity and Smoothnesscited by 6
  16. 2024To Shuffle or not to Shuffle: Auditing DP-SGD with Shufflingcited by 6
  17. 2024Privacy of the last iterate in cyclically-sampled DP-SGD on nonconvex composite lossescited by 5
  18. 2023On the Fairness Impacts of Private Ensembles Modelscited by 4
  19. 2023Tractable MCMC for Private Learning with Pure and Gaussian Differential Privacycited by 4
  20. 2024Characterizing the Training Dynamics of Private Fine-tuning with Langevin diffusioncited by 4
  21. 2025An Improved Privacy and Utility Analysis of Differentially Private SGD with Bounded Domain and Smooth Lossescited by 4
  22. 2025The Hitchhiker's Guide to Efficient, End-to-End, and Tight DP Auditingcited by 4
  23. 2026The Unseen Threat: Residual Knowledge in Machine Unlearning under Perturbed Samplescited by 4
  24. 2022Differentially Private Sampling from Rashomon Sets, and the Universality of Langevin Diffusion for Convex Optimizationcited by 3
  25. 2024Privacy of SGD under Gaussian or Heavy-Tailed Noise: Guarantees without Gradient Clippingcited by 3
  26. 2024Enhancing Privacy in Federated Learning through Local Trainingcited by 3
  27. 2025Empirical Privacy Variancecited by 3
  28. 2021Can Stochastic Gradient Langevin Dynamics Provide Differential Privacy for Deep Learning?cited by 2
  29. 2023Privacy Risk for anisotropic Langevin dynamics using relative entropy boundscited by 2
  30. 2023Locally Differentially Private Gradient Tracking for Distributed Online Learning over Directed Graphscited by 2
  31. 2024Tight Group-Level DP Guarantees for DP-SGD with Sampling via Mixture of Gaussians Mechanismscited by 2
  32. 2024Convex Approximation of Two-Layer ReLU Networks for Hidden State Differential Privacycited by 2
  33. 2025Generalization Bounds for Markov Algorithms through Entropy Flow Computationscited by 2
  34. 2025Statistical Inference for Differentially Private Stochastic Gradient Descentcited by 2
  35. 2022Recycling Scraps: Improving Private Learning by Leveraging Intermediate Checkpointscited by 1
  36. 2023Privacy Amplification by Iteration for ADMM with (Strongly) Convex Objective Functionscited by 1
  37. 2024Perturb-and-Project: Differentially Private Similarities and Marginalscited by 1
  38. 2026Harnessing Synthetic Data from Generative AI for Statistical Inferencecited by 1

50 most cited of 71 All 71 (JSON)