mlsanityData to 7 Oct 2026

DIFF2: Differential Private Optimization via Gradient Differences for Nonconvex Distributed Learning

arXiv 2302.03884 ↗First posted 8 Feb 2023cs.LG cs.CR math.OCcited by 10

Not on the map. Why?

Abstract (arXiv)

Differential private optimization for nonconvex smooth objective is considered. In the previous work, the best known utility bound is widetilde O(√d/(nvarepsilonDP)) in terms of the squared full gradient norm, which is achieved by Differential Private Gradient Descent (DP-GD) as an instance, where n is the sample size, d is the problem dimensionality and varepsilonDP is the differential privacy parameter. To improve the best known utility bound, we propose a new differential private optimization framework called DIFF2 (DIFFerential private optimization via gradient DIFFerences) that constructs a differential private global gradient estimator with possibly quite small variance based on communicated gradient differences rather than gradients themselves. It is shown that DIFF2 with a gradient descent subroutine achieves the utility of widetilde O(d2/3/(nvarepsilonDP)4/3), which can be significantly better than the previous one in terms of the dependence on the sample size n. To the best of our knowledge, this is the first fundamental result to improve the standard utility widetilde O(√d/(nvarepsilonDP)) for nonconvex objectives. Additionally, a more computational and communication efficient subroutine is combined with DIFF2 and its theoretical analysis is also given. Numerical experiments are conducted to validate the superiority of DIFF2 framework.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 42 of its 63 references that are in the map. First posted on arXiv 8 Feb 2023.

Builds on

42 of 63 references.

References · 42

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2016Deep Learning with Differential Privacycited by 2,682
  3. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  4. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  5. 2019Deep Leakage from Gradientscited by 1,004
  6. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 707
  7. 2017Renyi Differential Privacycited by 643
  8. 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 451
  9. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 401
  10. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
  11. 2018Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learningcited by 242
  12. 2019The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networkscited by 241
30 more
  1. 2018Subsampled R'enyi Differential Privacy and Analytical Moments Accountantcited by 234
  2. 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 218
  3. 2019Private Stochastic Convex Optimization with Optimal Ratescited by 183
  4. 2018cpSGD: Communication-efficient and differentially-private distributed SGDcited by 157
  5. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 152
  6. 2018Differentially Private Empirical Risk Minimization Revisited: Faster and More Generalcited by 127
  7. 2014Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Boundscited by 104
  8. 2021Private Stochastic Convex Optimization: Optimal Rates in ℓ1 Geometrycited by 83
  9. 2021Differentially Private Federated Learning on Heterogeneous Datacited by 74
  10. 2021Differential Privacy Dynamics of Langevin Diffusion and Noisy Gradient Descentcited by 71
  11. 2017Efficient Private ERM for Smooth Objectivescited by 61
  12. 2019Federated Learning with Bayesian Differential Privacycited by 60
  13. 2020On Differentially Private Stochastic Convex Optimization with Heavy-tailed Datacited by 51
  14. 2021Improved Rates for Differentially Private Stochastic Convex Optimization with Heavy-Tailed Datacited by 47
  15. 2021High Dimensional Differentially Private Stochastic Optimization with Heavy-tailed Datacited by 42
  16. 2018DP-ADMM: ADMM-based Distributed Learning with Differential Privacycited by 41
  17. 2022SoteriaFL: A Unified Framework for Private Federated Learning with Communication Compressioncited by 34
  18. 2021Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learningcited by 30
  19. 2021Dynamic Differential-Privacy Preserving SGDcited by 29
  20. 2019Efficient Privacy-Preserving Stochastic Nonconvex Optimizationcited by 27
  21. 2020Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Boundscited by 26
  22. 2022Beyond Uniform Lipschitz Condition in Differentially Private Optimizationcited by 19
  23. 2020Private and Communication-Efficient Edge Learning: A Sparse Differential Gaussian-Masking Distributed SGD Approachcited by 18
  24. 2019DP-LSSGD: A Stochastic Optimization Method to Lift the Utility in Privacy-Preserving ERMcited by 15
  25. 2020Differentially Private Accelerated Optimization Algorithmscited by 13
  26. 2022Differentially private Riemannian optimizationcited by 12
  27. 2022Momentum Aggregation for Private Non-convex ERMcited by 10
  28. 2020Differentially Private Federated Learning with Laplacian Smoothingcited by 8
  29. 2020Differentially Private ADMM for Convex Distributed Learning: Improved Accuracy via Multi-Step Approximationcited by 8
  30. 2020Differentially Private ADMM Algorithms for Machine Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 4; 2026: 1.

10 papers.

  1. 2023Convergence and Privacy of Decentralized Nonconvex Optimization with Gradient Clipping and Communication Compressioncited by 9
  2. 2025Improved Rates of Differentially Private Nonconvex-Strongly-Concave Minimax Optimizationcited by 5
  3. 2023Differentially Private Non-Convex Optimization under the KL Condition with Optimal Ratescited by 4
  4. 2024DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learningcited by 4
  5. 2024Privacy of SGD under Gaussian or Heavy-Tailed Noise: Guarantees without Gradient Clippingcited by 3
  6. 2025Second-Order Convergence in Private Stochastic Non-Convex Optimizationcited by 3
  7. 2023Generalizing Differentially Private Decentralized Deep Learning with Multi-Agent Consensuscited by 1
  8. 2025Smoothed Normalization for Efficient Distributed Private Optimizationcited by 1
  9. 2025Personalized Federated Training of Diffusion Models with Privacy Guaranteescited by 0
  10. 2026Private Decentralized Optimization with Noise Reduction and Bias Correctioncited by 0