mlsanityData to 7 Oct 2026

Private Adaptive Gradient Methods for Convex Optimization

arXiv 2106.13756 ↗First posted 25 Jun 2021cs.LG cs.CR math.OCcited by 46

Not on the map. Why?

Abstract (arXiv)

We study adaptive methods for differentially private convex optimization, proposing and analyzing differentially private variants of a Stochastic Gradient Descent (SGD) algorithm with adaptive stepsizes, as well as the AdaGrad algorithm. We provide upper bounds on the regret of both algorithms and show that the bounds are (worst-case) optimal. As a consequence of our development, we show that our private versions of AdaGrad outperform adaptive SGD, which in turn outperforms traditional SGD in scenarios with non-isotropic gradients where (non-private) Adagrad provably outperforms SGD. The major challenge is that the isotropic noise typically added for privacy dominates the signal in gradient geometry for high-dimensional problems; approaches to this that effectively optimize over lower-dimensional subspaces simply ignore the actual problems that varying gradient geometries introduce. In contrast, we study non-isotropic clipping and noise addition, developing a principled theoretical approach; the consequent procedures also enjoy significantly stronger empirical performance than prior approaches.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Builds on 13 of its 34 references that are in the map. First posted on arXiv 25 Jun 2021.

Builds on

13 of 34 references.

References · 13

  1. 2016Pointer Sentinel Mixture Modelscited by 2,992
  2. 2016Deep Learning with Differential Privacycited by 2,682
  3. 2019Private Stochastic Convex Optimization with Optimal Ratescited by 183
  4. 2020Private Stochastic Convex Optimization: Optimal Rates in Linear Timecited by 152
  5. 2014Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Boundscited by 104
  6. 2016Minimax Optimal Procedures for Locally Private Estimationcited by 103
  7. 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 88
  8. 2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
  9. 2021Private Stochastic Convex Optimization: Optimal Rates in ℓ1 Geometrycited by 83
  10. 2019AdaCliP: Adaptive Clipping for Private SGDcited by 77
  11. 2018Stochastic (Approximate) Proximal Point Methods: Convergence, Optimality, and Adaptivitycited by 61
  12. 2020Fast Dimension Independent Private AdaGrad on Publicly Estimated Subspacescited by 10
1 more
  1. 2019Geometry, Computation, and Optimality in Stochastic Optimizationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 20; 2023: 6; 2024: 7; 2025: 4; 2026: 5.

46 papers.

Sort:

  1. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  2. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  3. 2022Exploring the Limits of Differentially Private Deep Learning with Group-wise Clippingcited by 52
  4. 2021Public Data-Assisted Mirror Descent for Private Model Trainingcited by 49
  5. 2022When Does Differentially Private Learning Not Suffer in High Dimensions?cited by 48
  6. 2022Differentially Private Learning Needs Hidden State (Or Much Faster Convergence)cited by 45
  7. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 40
  8. 2022Private Adaptive Optimization with Side Informationcited by 38
  9. 2023Why Is Public Pretraining Necessary for Private Model Training?cited by 36
  10. 2022Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimizationcited by 35
  11. 2022FLAIR: Federated Learning Annotated Image Repositorycited by 22
  12. 2022Differentially Private Learning with Per-Sample Adaptive Clippingcited by 21
34 more
  1. 2021Adapting to Function Difficulty and Growth Conditions in Private Optimizationcited by 19
  2. 2022Beyond Uniform Lipschitz Condition in Differentially Private Optimizationcited by 19
  3. 2022Differentially Private Adaptive Optimization with Delayed Preconditionerscited by 18
  4. 2022Private Online Prediction from Experts: Separations and Faster Ratescited by 17
  5. 2023Fast Optimal Locally Private Mean Estimation via Random Projectionscited by 14
  6. 2021Faster Rates of Private Stochastic Convex Optimizationcited by 12
  7. 2022Differentially private Riemannian optimizationcited by 12
  8. 2023User-level Differentially Private Stochastic Convex Optimization: Efficient Algorithms with Optimal Ratescited by 12
  9. 2022Momentum Aggregation for Private Non-convex ERMcited by 10
  10. 2023PILLAR: How to make semi-private learning more effectivecited by 10
  11. 2022Stability Based Generalization Bounds for Exponential Family Langevin Dynamicscited by 8
  12. 2022Differentially Private ℓ1-norm Linear Regression with Heavy-tailed Datacited by 8
  13. 2022Mixed Federated Learning: Joint Decentralized and Centralized Learningcited by 8
  14. 2023Spectral-DP: Differentially Private Deep Learning through Spectral Perturbation and Filteringcited by 6
  15. 2024Private Stochastic Convex Optimization with Heavy Tails: Near-Optimality from Simple Reductionscited by 6
  16. 2022Differentially Private Online-to-Batch for Smooth Lossescited by 5
  17. 2024Delving into Differentially Private Transformercited by 5
  18. 2022Differentially Private Deep Learning with ModelMixcited by 4
  19. 2024DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learningcited by 4
  20. 2022Private Stochastic Optimization With Large Worst-Case Lipschitz Parametercited by 3
  21. 2022Private optimization in the interpolation regime: faster rates and hardness resultscited by 3
  22. 2025Nearly Optimal Differentially Private ReLU Regressioncited by 3
  23. 2025Beyond Ordinary Lipschitz Constraints: Differentially Private Stochastic Optimization with Tsybakov Noise Conditioncited by 2
  24. 2025Private Zeroth-Order Optimization with Public Datacited by 2
  25. 2025Understanding Private Learning From Feature Perspectivecited by 2
  26. 2024Federated Online Prediction from Experts with Differential Privacy: Separations and Regret Speed-upscited by 1
  27. 2024Noise is All You Need: Private Second-Order Convergence of Noisy SGDcited by 1
  28. 2024Enhancing DP-SGD through Non-monotonous Adaptive Scaling Gradient Weightcited by 1
  29. 2026Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspectivecited by 1
  30. 2024Training with Differential Privacy: A Gradient-Preserving Noise Reduction Approach with Provable Securitycited by 0
  31. 2026Understanding the Impact of Differentially Private Training on Memorization of Long-Tailed Datacited by 0
  32. 2026Optimal Rates for Pure varepsilon-Differentially Private Stochastic Convex Optimization with Heavy Tailscited by 0
  33. 2026Integrating Feature Correlation in Differential Privacy with Applications in DP-ERMcited by 0
  34. 2026Deep Learning under Fractional-Order Differential Privacycited by 0