mlsanityData to 7 Oct 2026

Convergent Privacy Loss of Noisy-SGD without Convexity and Smoothness

arXiv 2410.01068 ↗First posted 1 Oct 2024cs.LG cs.CRcited by 6

Not on the map. Why?

Abstract (arXiv)

We study the Differential Privacy (DP) guarantee of hidden-state Noisy-SGD algorithms over a bounded domain. Standard privacy analysis for Noisy-SGD assumes all internal states are revealed, which leads to a divergent R'enyi DP bound with respect to the number of iterations. Ye & Shokri (2022) and Altschuler & Talwar (2022) proved convergent bounds for smooth (strongly) convex losses, and raise open questions about whether these assumptions can be relaxed. We provide positive answers by proving convergent R'enyi DP bound for non-convex non-smooth losses, where we show that requiring losses to have H"older continuous gradient is sufficient. We also provide a strictly better privacy bound compared to state-of-the-art results for smooth strongly convex losses. Our analysis relies on the improvement of shifted divergence analysis in multiple aspects, including forward Wasserstein distance tracking, identifying the optimal shifts allocation, and the H"older reduction lemma. Our results further elucidate the benefit of hidden-state analysis for DP and its applicability.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 18 of its 22 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

18 of 22 references.

References · 18

  1. 2016Deep Learning with Differential Privacycited by 2,682
  2. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  3. 2013The Composition Theorem for Differential Privacycited by 290
  4. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 199
  5. 2021Numerical Composition of Differential Privacycited by 174
  6. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 169
  7. 2018Privacy Amplification by Iterationcited by 97
  8. 2021Differential Privacy Dynamics of Langevin Diffusion and Noisy Gradient Descentcited by 71
  9. 2022Privacy of Noisy Stochastic Gradient Descent: More Iterations without More Privacy Losscited by 60
  10. 2024Langevin Unlearning: A New Perspective of Noisy Gradient Descent for Machine Unlearningcited by 48
  11. 2022Differentially Private Learning Needs Hidden State (Or Much Faster Convergence)cited by 45
  12. 2024How Private are DP-SGD Implementations?cited by 22
6 more
  1. 2022Resolving the Mixing Time of the Langevin Algorithm to its Stationary Distribution for Log-Concave Samplingcited by 21
  2. 2023DP-SGD Without Clipping: The Lipschitz Neural Network Waycited by 11
  3. 2023Privacy Loss of Noisy Stochastic Gradient Descent Might Converge Even for Non-Convex Lossescited by 10
  4. 2024It's Our Loss: No Privacy Amplification for Hidden State DP-SGD With Non-Convex Losscited by 10
  5. 2024Privacy of the last iterate in cyclically-sampled DP-SGD on nonconvex composite lossescited by 5
  6. 2024Differentially Private Graph Diffusion with Applications in Personalized PageRankscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025An Improved Privacy and Utility Analysis of Differentially Private SGD with Bounded Domain and Smooth Lossescited by 4
  2. 2025Empirical Privacy Variancecited by 3
  3. 2025Mixing Times and Privacy Analysis for the Projected Langevin Algorithm under a Modulus of Continuitycited by 0
  4. 2025Privacy Amplification in Differentially Private Zeroth-Order Optimization with Hidden Statescited by 0
  5. 2025Descend or Rewind? Stochastic Gradient Descent Unlearningcited by 0
  6. 2026Challenges in Enabling Private Data Valuationcited by 0