mlsanityData to 14 Oct 2026

Momentum Aggregation for Private Non-convex ERM

arXiv 2210.06328 ↗First posted 12 Oct 2022cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

We introduce new algorithms and convergence guarantees for privacy-preserving non-convex Empirical Risk Minimization (ERM) on smooth d-dimensional objectives. We develop an improved sensitivity analysis of stochastic gradient descent on smooth objectives that exploits the recurrence of examples in different epochs. By combining this new approach with recent analysis of momentum with private aggregation techniques, we provide an (ε,δ)-differential private algorithm that finds a gradient of norm tilde Oleft(fracd1/3(ε N)2/3right) in Oleft(fracN7/3ε4/3d2/3right) gradient evaluations, improving the previous best gradient bound of tilde Oleft(fracd1/4√ε Nright).

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 28 of its 53 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

28 of 53 references.

References · 28

  1. 2016Deep Learning with Differential Privacycited by 2,728
  2. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  3. 2017Renyi Differential Privacycited by 650
  4. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 307
  5. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  6. 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 287
  7. 2018Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergencescited by 242
  8. 2019Private Stochastic Convex Optimization with Optimal Ratescited by 186
  9. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 170
  10. 2020Private Stochastic Convex Optimization: Optimal Rates in Linear Timecited by 153
  11. 2020Momentum Improves Normalized SGDcited by 139
  12. 2018Differentially Private Empirical Risk Minimization Revisited: Faster and More Generalcited by 128
16 more
  1. 2020Random Reshuffling: Simple Analysis with Vast Improvementscited by 115
  2. 2015Strongly Adaptive Online Learningcited by 107
  3. 2018Privacy Amplification by Iterationcited by 100
  4. 2018Stochastic Nested Variance Reduction for Nonconvex Optimizationcited by 63
  5. 2017Efficient Private ERM for Smooth Objectivescited by 61
  6. 2021Non-Euclidean Differentially Private Stochastic Convex Optimization: Optimal Rates in Linear Timecited by 51
  7. 2021Private Adaptive Gradient Methods for Convex Optimizationcited by 46
  8. 2021Differentially Private Stochastic Optimization: New Results in Convex and Non-Convex Settingscited by 45
  9. 2020Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitationscited by 40
  10. 2019Efficient Privacy-Preserving Stochastic Nonconvex Optimizationcited by 27
  11. 2020Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Boundscited by 26
  12. 2014Private Empirical Risk Minimization Beyond the Worst Case: The Effect of the Constraint Set Geometrycited by 24
  13. 2021Private Non-smooth Empirical Risk Minimization and Stochastic Convex Optimization in Subquadratic Stepscited by 21
  14. 2018Tight Dimension Independent Lower Bound on the Expected Convergence Rate for Diminishing Step Sizes in SGDcited by 18
  15. 2021Better SGD using Second-order Momentumcited by 15
  16. 2021Applications of artificial intelligence in drug development using real-world datacited by 7

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 5; 2025: 1.

10 papers.

  1. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  2. 2023Gradient Descent with Linearly Correlated Noise: Theory and Applications to Differential Privacycited by 18
  3. 2023Private (Stochastic) Non-Convex Optimization Revisited: Second-Order Stationary Points and Excess Riskscited by 12
  4. 2023DIFF2: Differential Private Optimization via Gradient Differences for Nonconvex Distributed Learningcited by 10
  5. 2024How to Make the Gradients Small Privately: Improved Rates for Differentially Private Non-Convex Optimizationcited by 10
  6. 2024Private Zeroth-Order Nonsmooth Nonconvex Optimizationcited by 8
  7. 2025Improved Rates of Differentially Private Nonconvex-Strongly-Concave Minimax Optimizationcited by 5
  8. 2024Reevaluating Theoretical Analysis Methods for Optimization in Deep Learningcited by 3
  9. 2024Differentially Private Bilevel Optimizationcited by 2
  10. 2024Adaptive Batch Size for Privately Finding Second-Order Stationary Pointscited by 2