mlsanityData to 7 Oct 2026

Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimization

arXiv 2206.13033 ↗First posted 27 Jun 2022cs.LG cs.IT math.ITcited by 35

Not on the map. Why?

Abstract (arXiv)

By ensuring differential privacy in the learning algorithms, one can rigorously mitigate the risk of large models memorizing sensitive training data. In this paper, we study two algorithms for this purpose, i.e., DP-SGD and DP-NSGD, which first clip or normalize textitper-sample gradients to bound the sensitivity and then add noise to obfuscate the exact information. We analyze the convergence behavior of these two algorithms in the non-convex optimization setting with two common assumptions and achieve a rate Oleft(√[4]fracdlog(1/δ)N2ε2right) of the gradient norm for a d-dimensional model, N samples and (ε,δ)-DP, which improves over previous bounds under much weaker assumptions. Specifically, we introduce a regularizing factor in DP-NSGD and show that it is crucial in the convergence proof and subtly controls the bias and noise trade-off. Our proof deliberately handles the per-sample gradient clipping and normalization that are specified for the private setting. Empirically, we demonstrate that these two algorithms achieve similar best accuracy while DP-NSGD is comparatively easier to tune than DP-SGD and hence may help further save the privacy budget when accounting the tuning effort.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 36 of its 52 references that are in the map. First posted on arXiv 27 Jun 2022.

Builds on

36 of 52 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2016Deep Learning with Differential Privacycited by 2,682
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2018Group Normalizationcited by 1,514
  8. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  9. 2017Renyi Differential Privacycited by 643
  10. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  11. 2016Concentrated Differential Privacy: Simplifications, Extensions, and Lower Boundscited by 339
  12. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
24 more
  1. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 303
  2. 2018Subsampled R'enyi Differential Privacy and Analytical Moments Accountantcited by 234
  3. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 233
  4. 2019Differentially Private Learning with Adaptive Clippingcited by 218
  5. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 199
  6. 2021Numerical Composition of Differential Privacycited by 174
  7. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 140
  8. 2020Momentum Improves Normalized SGDcited by 136
  9. 2018Differentially Private Empirical Risk Minimization Revisited: Faster and More Generalcited by 127
  10. 2020Tempered Sigmoid Activations for Deep Learning with Differential Privacycited by 121
  11. 2019Deep Learning with Gaussian Differential Privacycited by 112
  12. 2014Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Boundscited by 104
  13. 2021Hyperparameter Tuning with Renyi Differential Privacycited by 104
  14. 2020Improved Analysis of Clipping Algorithms for Non-convex Optimizationcited by 88
  15. 2021High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tailscited by 80
  16. 2019AdaCliP: Adaptive Clipping for Private SGDcited by 77
  17. 2021Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacycited by 66
  18. 2019The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descentcited by 46
  19. 2021Private Adaptive Gradient Methods for Convex Optimizationcited by 46
  20. 2021Non-convex Distributionally Robust Optimization: Non-asymptotic Analysiscited by 39
  21. 2020Dual Policy Distillationcited by 25
  22. 2021The Role of Adaptive Optimizers for Honest Private Hyperparameter Selectioncited by 25
  23. 2021Differentially Private SGD with Non-Smooth Lossescited by 22
  24. 2020Differentially Private Accelerated Optimization Algorithmscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 18; 2024: 8; 2025: 4; 2026: 3.

35 papers.

Sort:

By area:

  1. 2023Revisiting Gradient Clipping: Stochastic bias and tight convergence guaranteescited by 83
  2. 2023Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptionscited by 55
  3. 2022Differentially Private Bias-Term Fine-tuning of Foundation Modelscited by 47
  4. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  5. 2023Variance-reduced Clipping for Non-convex Optimizationcited by 22
  6. 2022Differentially Private Learning with Per-Sample Adaptive Clippingcited by 21
  7. 2023Practical Differentially Private and Byzantine-resilient Federated Learningcited by 18
  8. 2023Tangent Transformers for Composition, Privacy and Removalcited by 13
  9. 2023Differentially Private SGD Without Clipping Bias: An Error-Feedback Approachcited by 13
  10. 2023Private (Stochastic) Non-Convex Optimization Revisited: Second-Order Stationary Points and Excess Riskscited by 12
  11. 2023DP-SGD Without Clipping: The Lipschitz Neural Network Waycited by 11
  12. 2023Convergence and Privacy of Decentralized Nonconvex Optimization with Gradient Clipping and Communication Compressioncited by 9
23 more
  1. 2024Pre-training Differentially Private Models with Limited Public Datacited by 8
  2. 2024DOPPLER: Differentially Private Optimizers with Low-pass Filter for Privacy Noise Reductioncited by 8
  3. 2023Safeguarding Data in Multimodal AI: A Differentially Private Approach to CLIP Trainingcited by 7
  4. 2023High-probability Convergence Bounds for Nonlinear Stochastic Gradient Descent Under Heavy-tailed Noisecited by 7
  5. 2023Zero redundancy distributed learning with differential privacycited by 7
  6. 2025Towards hyperparameter-free optimization with differential privacycited by 7
  7. 2025DC-SGD: Differentially Private SGD with Dynamic Clipping through Gradient Norm Distribution Estimationcited by 6
  8. 2024Delving into Differentially Private Transformercited by 5
  9. 2024Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tailscited by 4
  10. 2024DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learningcited by 4
  11. 2023DPFormer: Learning Differentially Private Transformer on Long-Tailed Datacited by 2
  12. 2023PCDP-SGD: Improving the Convergence of Differentially Private SGD via Projection in Advancecited by 2
  13. 2023ALI-DPFL: Differentially Private Federated Learning with Adaptive Local Iterationscited by 1
  14. 2023The Relative Gaussian Mechanism and its Application to Private Gradient Descentcited by 1
  15. 2024Noise is All You Need: Private Second-Order Convergence of Noisy SGDcited by 1
  16. 2024Nonlinear Stochastic Gradient Descent and Heavy-tailed Noise: A Unified Framework and High-probability Guaranteescited by 1
  17. 2024Enhancing DP-SGD through Non-monotonous Adaptive Scaling Gradient Weightcited by 1
  18. 2025Smoothed Normalization for Efficient Distributed Private Optimizationcited by 1
  19. 2023DP-SGD-Global-Adapt-V2-S: Triad Improvements of Privacy, Accuracy and Fairness via Step Decay Noise Multiplier and Step Decay Upper Clipping Thresholdcited by 0
  20. 2025Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)cited by 0
  21. 2026Provably Convergent Decentralized Optimization over Directed Graphs under Generalized Smoothnesscited by 0
  22. 2026Mitigating Membership Inference in Intermediate Representations with Differentially Private Trainingcited by 0
  23. 2026From Gradient Clipping to Structural Refinement: Improving DPSGD for Medical Image Segmentationcited by 0