mlsanityData to 7 Oct 2026

Beyond Uniform Lipschitz Condition in Differentially Private Optimization

arXiv 2206.10713 ↗First posted 21 Jun 2022cs.LG stat.MLcited by 19

Not on the map. Why?

Abstract (arXiv)

Most prior results on differentially private stochastic gradient descent (DP-SGD) are derived under the simplistic assumption of uniform Lipschitzness, i.e., the per-sample gradients are uniformly bounded. We generalize uniform Lipschitzness by assuming that the per-sample gradients have sample-dependent upper bounds, i.e., per-sample Lipschitz constants, which themselves may be unbounded. We provide principled guidance on choosing the clip norm in DP-SGD for convex over-parameterized settings satisfying our general version of Lipschitzness when the per-sample Lipschitz constants are bounded; specifically, we recommend tuning the clip norm only till values up to the minimum per-sample Lipschitz constant. This finds application in the private training of a softmax layer on top of a deep network pre-trained on public data. We verify the efficacy of our recommendation via experiments on 8 datasets. Furthermore, we provide new convergence results for DP-SGD on convex and nonconvex functions when the Lipschitz constants are unbounded but have bounded moments, i.e., they are heavy-tailed.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 31 of its 70 references that are in the map. First posted on arXiv 21 Jun 2022.

Builds on

31 of 70 references.

References · 31

  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  2. 2016Deep Learning with Differential Privacycited by 2,682
  3. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
  4. 2016Concentrated Differential Privacy: Simplifications, Extensions, and Lower Boundscited by 339
  5. 2019Differentially Private Learning with Adaptive Clippingcited by 218
  6. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  7. 2019Private Stochastic Convex Optimization with Optimal Ratescited by 183
  8. 2020Private Stochastic Convex Optimization: Optimal Rates in Linear Timecited by 152
  9. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  10. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 140
  11. 2018Differentially Private Empirical Risk Minimization Revisited: Faster and More Generalcited by 127
  12. 2020Tempered Sigmoid Activations for Deep Learning with Differential Privacycited by 121
19 more
  1. 2021Hyperparameter Tuning with Renyi Differential Privacycited by 104
  2. 2021Private Stochastic Convex Optimization: Optimal Rates in ℓ1 Geometrycited by 83
  3. 2021Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacycited by 66
  4. 2017Efficient Private ERM for Smooth Objectivescited by 61
  5. 2020Differentially Private Assouad, Fano, and Le Camcited by 54
  6. 2020On Differentially Private Stochastic Convex Optimization with Heavy-tailed Datacited by 51
  7. 2021Improved Rates for Differentially Private Stochastic Convex Optimization with Heavy-Tailed Datacited by 47
  8. 2021Private Adaptive Gradient Methods for Convex Optimizationcited by 46
  9. 2021Differentially Private Stochastic Optimization: New Results in Convex and Non-Convex Settingscited by 45
  10. 2015From error bounds to the complexity of first-order descent methods for convex functionscited by 42
  11. 2021High Dimensional Differentially Private Stochastic Optimization with Heavy-tailed Datacited by 42
  12. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 40
  13. 2018Subgradient methods for sharp weakly convex functionscited by 33
  14. 2022Faster Rates of Convergence to Stationary Points in Differentially Private Optimizationcited by 32
  15. 2021Dynamic Differential-Privacy Preserving SGDcited by 29
  16. 2019Efficient Privacy-Preserving Stochastic Nonconvex Optimizationcited by 27
  17. 2014Private Empirical Risk Minimization Beyond the Worst Case: The Effect of the Constraint Set Geometrycited by 24
  18. 2021Private Non-smooth Empirical Risk Minimization and Stochastic Convex Optimization in Subquadratic Stepscited by 21
  19. 2021Adaptive Differentially Private Empirical Risk Minimizationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 6; 2024: 3; 2025: 4; 2026: 3.

19 papers.

Sort:

By area:

  1. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  2. 2023Revisiting Gradient Clipping: Stochastic bias and tight convergence guaranteescited by 83
  3. 2023Gradient Descent with Linearly Correlated Noise: Theory and Applications to Differential Privacycited by 18
  4. 2023DIFF2: Differential Private Optimization via Gradient Differences for Nonconvex Distributed Learningcited by 10
  5. 2023Convergence and Privacy of Decentralized Nonconvex Optimization with Gradient Clipping and Communication Compressioncited by 9
  6. 2024BadGD: A unified data-centric framework to identify gradient descent vulnerabilitiescited by 5
  7. 2025Differentially Private Federated Learning With Time-Adaptive Privacy Spendingcited by 5
  8. 2022Private Stochastic Optimization With Large Worst-Case Lipschitz Parametercited by 3
  9. 2023Towards Understanding Generalization and Stability Gaps between Centralized and Decentralized Federated Learningcited by 3
  10. 2024Data Deletion for Linear Regression with Noisy SGDcited by 2
  11. 2024Differential Private Stochastic Optimization with Heavy-tailed Data: Towards Optimal Ratescited by 1
  12. 2025Scaled Supervision is an Implicit Lipschitz Regularizercited by 1
7 more
  1. 2026Stability beyond Bounded Differences: Sharp Generalization Bounds under Finite Lp Momentscited by 1
  2. 2022Efficient Private SCO for Heavy-Tailed Data via Averaged Clippingcited by 0
  3. 2022Private Blind Model Averaging - Distributed, Non-interactive, and Convergentcited by 0
  4. 2025On Theoretical Limits of Learning with Label Differential Privacycited by 0
  5. 2025Balancing Utility and Privacy: Dynamically Private SGD with Random Projectioncited by 0
  6. 2026Amplified Patch-Level Differential Privacy for Free via Random Croppingcited by 0
  7. 2026Rethinking the Personalized Relaxed Initialization in the Federated Learning: Consistency and Generalizationcited by 0