mlsanityData to 14 Oct 2026

Differentially Private Non-convex Learning for Multi-layer Neural Networks

arXiv 2310.08425 ↗First posted 12 Oct 2023cs.LG cs.CR stat.MLcited by 10

Not on the map. Why?

Abstract (arXiv)

This paper focuses on the problem of Differentially Private Stochastic Optimization for (multi-layer) fully connected neural networks with a single output node. In the first part, we examine cases with no hidden nodes, specifically focusing on Generalized Linear Models (GLMs). We investigate the well-specific model where the random noise possesses a zero mean, and the link function is both bounded and Lipschitz continuous. We propose several algorithms and our analysis demonstrates the feasibility of achieving an excess population risk that remains invariant to the data dimension. We also delve into the scenario involving the ReLU link function, and our findings mirror those of the bounded link function. We conclude this section by contrasting well-specified and misspecified models, using ReLU regression as a representative example. In the second part of the paper, we extend our ideas to two-layer neural networks with sigmoid or ReLU activation functions in the well-specified model. In the third part, we study the theoretical guarantees of DP-SGD in Abadi et al. (2016) for fully connected multi-layer neural networks. By utilizing recent advances in Neural Tangent Kernel theory, we provide the first excess population risk when both the sample size and the width of the network are sufficiently large. Additionally, we discuss the role of some parameters in DP-SGD regarding their utility, both theoretically and empirically.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 24 of its 48 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

24 of 48 references.

References · 24

  1. 2016Deep Learning with Differential Privacycited by 2,728
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  5. 2017Deep Neural Networks as Gaussian Processescited by 622
  6. 2014Convex Optimization: Algorithms and Complexitycited by 575
  7. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  8. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  9. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 222
  10. 2020Private Stochastic Convex Optimization: Optimal Rates in Linear Timecited by 153
  11. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
  12. 2018Differentially Private Empirical Risk Minimization Revisited: Faster and More Generalcited by 128
12 more
  1. 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 90
  2. 2019Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networkscited by 60
  3. 2022When Does Differentially Private Learning Not Suffer in High Dimensions?cited by 48
  4. 2020Agnostic Learning of a Single Neuron with Gradient Descentcited by 46
  5. 2021Differentially Private Stochastic Optimization: New Results in Convex and Non-Convex Settingscited by 45
  6. 2021High Dimensional Differentially Private Stochastic Optimization with Heavy-tailed Datacited by 42
  7. 2020Approximation Schemes for ReLU Regressioncited by 34
  8. 2019Time/Accuracy Tradeoffs for Learning a ReLU with respect to Gaussian Marginalscited by 32
  9. 2022Faster Rates of Convergence to Stationary Points in Differentially Private Optimizationcited by 32
  10. 2019Efficient Privacy-Preserving Stochastic Nonconvex Optimizationcited by 27
  11. 2022Differentially Private Generalized Linear Models Revisitedcited by 17
  12. 2019Weighted Distributed Differential Privacy ERM: Convex and Non-convexcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 2; 2026: 1.

10 papers.

  1. 2024Prompt-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compressioncited by 18
  2. 2023Unmasking Transformers: A Theoretical Approach to Data Recovery via Attention Weightscited by 12
  3. 2024Dialectical Alignment: Resolving the Tension of 3H and Security Threats of LLMscited by 11
  4. 2024How to Make the Gradients Small Privately: Improved Rates for Differentially Private Non-Convex Optimizationcited by 10
  5. 2024MONAL: Model Autophagy Analysis for Modeling Human-AI Interactionscited by 7
  6. 2023A Theoretical Insight into Attack and Defense of Gradient Leakage in Transformercited by 6
  7. 2024Revisiting Differentially Private Hyper-parameter Tuningcited by 5
  8. 2025Nearly Optimal Differentially Private ReLU Regressioncited by 3
  9. 2025Differentially Private Sparse Linear Regression with Heavy-tailed Responsescited by 1
  10. 2026Understanding the Impact of Differentially Private Training on Memorization of Long-Tailed Datacited by 1