mlsanityData to 7 Oct 2026

Adaptive Differentially Private Empirical Risk Minimization

arXiv 2110.07435 ↗First posted 14 Oct 2021cs.LG eess.IV math.OCcited by 5

Not on the map. Why?

Abstract (arXiv)

We propose an adaptive (stochastic) gradient perturbation method for differentially private empirical risk minimization. At each iteration, the random noise added to the gradient is optimally adapted to the stepsize; we name this process adaptive differentially private (ADP) learning. Given the same privacy budget, we prove that the ADP method considerably improves the utility guarantee compared to the standard differentially private method in which vanilla random noise is added. Our method is particularly useful for gradient-based algorithms with time-varying learning rates, including variants of AdaGrad (Duchi et al., 2011). We provide extensive numerical experiments to demonstrate the effectiveness of the proposed adaptive differentially private algorithm.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 34 of its 58 references that are in the map. First posted on arXiv 14 Oct 2021.

Builds on

34 of 58 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2016Deep Learning with Differential Privacycited by 2,682
  5. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  8. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  9. 2020Adaptive Federated Optimizationcited by 1,036
  10. 2019On the Convergence of Adam and Beyondcited by 834
  11. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  12. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
22 more
  1. 2013The Composition Theorem for Differential Privacycited by 290
  2. 2019Differentially Private Learning with Adaptive Clippingcited by 218
  3. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 199
  4. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  5. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 173
  6. 2020Stability of Stochastic Gradient Descent on Nonsmooth Convex Lossescited by 172
  7. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 155
  8. 2020Private Stochastic Convex Optimization: Optimal Rates in Linear Timecited by 152
  9. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 140
  10. 2018Differentially Private Empirical Risk Minimization Revisited: Faster and More Generalcited by 127
  11. 2019Deep Learning with Gaussian Differential Privacycited by 112
  12. 2020Revisiting Membership Inference Under Realistic Assumptionscited by 108
  13. 2014Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Boundscited by 104
  14. 2020Exactly Computing the Local Lipschitz Constant of ReLU Networkscited by 102
  15. 2019AdaCliP: Adaptive Clipping for Private SGDcited by 77
  16. 2019The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squarescited by 67
  17. 2017Efficient Private ERM for Smooth Objectivescited by 61
  18. 2018Online Adaptive Methods, Universality and Accelerationcited by 51
  19. 2018Concentrated Differentially Private Gradient Descent with Adaptive per-Iteration Privacy Budgetcited by 31
  20. 2020Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Boundscited by 26
  21. 2021Membership Inference Attacks on Deep Regression Models for Neuroimagingcited by 19
  22. 2019DP-LSSGD: A Stochastic Optimization Method to Lift the Utility in Privacy-Preserving ERMcited by 15

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 2; 2023: 0; 2024: 1.

5 papers.

  1. 2021Dynamic Differential-Privacy Preserving SGDcited by 29
  2. 2022Differentially Private Learning with Per-Sample Adaptive Clippingcited by 21
  3. 2022Beyond Uniform Lipschitz Condition in Differentially Private Optimizationcited by 19
  4. 2021DP-FP: Differentially Private Forward Propagation for Large Modelscited by 6
  5. 2024Enhancing DP-SGD through Non-monotonous Adaptive Scaling Gradient Weightcited by 1