mlsanityData to 14 Oct 2026

Differentially Private Deep Learning with ModelMix

arXiv 2210.03843 ↗First posted 7 Oct 2022cs.LG cs.CRcited by 4

Not on the map. Why?

Abstract (arXiv)

Training large neural networks with meaningful/usable differential privacy security guarantees is a demanding challenge. In this paper, we tackle this problem by revisiting the two key operations in Differentially Private Stochastic Gradient Descent (DP-SGD): 1) iterative perturbation and 2) gradient clipping. We propose a generic optimization framework, called em ModelMix, which performs random aggregation of intermediate model states. It strengthens the composite privacy analysis utilizing the entropy of the training trajectory and improves the (ε, δ) DP security parameters by an order of magnitude. We provide rigorous analyses for both the utility guarantees and privacy amplification of ModelMix. In particular, we present a formal study on the effect of gradient clipping in DP-SGD, which provides theoretical instruction on how hyper-parameters should be selected. We also introduce a refined gradient clipping method, which can further sharpen the privacy loss in private learning when combined with ModelMix. Thorough experiments with significant privacy/utility improvement are presented to support our theory. We train a Resnet-20 network on CIFAR10 with 70.4% accuracy via ModelMix given (ε=8, δ=10-5) DP-budget, compared to the same performance but with (ε=145.8,δ=10-5) using regular DP-SGD; assisted with additional public low-dimensional gradient embedding, one can further improve the accuracy to 79.1% with (ε=6.1, δ=10-5) DP-budget, compared to the same performance but with (ε=111.2, δ=10-5) without ModelMix.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 40 of its 65 references that are in the map. First posted on arXiv 7 Oct 2022.

Builds on

40 of 65 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  4. 2016Deep Learning with Differential Privacycited by 2,728
  5. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  6. 2017Random Erasing Data Augmentationcited by 1,239
  7. 2019Deep Leakage from Gradientscited by 1,021
  8. 2017Renyi Differential Privacycited by 650
  9. 2018How Does Batch Normalization Help Optimization?cited by 357
  10. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 348
  11. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  12. 2017How to Escape Saddle Points Efficientlycited by 285
28 more
  1. 2018Scalable Private Learning with PATEcited by 269
  2. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 243
  3. 2018Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergencescited by 242
  4. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 235
  5. 2019Differentially Private Learning with Adaptive Clippingcited by 220
  6. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 218
  7. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 204
  8. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 199
  9. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 142
  10. 2018Differentially Private Empirical Risk Minimization Revisited: Faster and More Generalcited by 128
  11. 2020Tempered Sigmoid Activations for Deep Learning with Differential Privacycited by 122
  12. 2014Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Boundscited by 107
  13. 2018Privacy Amplification by Iterationcited by 100
  14. 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 90
  15. 2020InstaHide: Instance-hiding Schemes for Private Distributed Learningcited by 90
  16. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 88
  17. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  18. 2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
  19. 2017Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpointscited by 82
  20. 2018Private Selection from Private Candidatescited by 75
  21. 2017Non-convex Finite-Sum Optimization Via SCSG Methodscited by 61
  22. 2020On Differentially Private Stochastic Convex Optimization with Heavy-tailed Datacited by 51
  23. 2021Private Adaptive Gradient Methods for Convex Optimizationcited by 46
  24. 2021High Dimensional Differentially Private Stochastic Optimization with Heavy-tailed Datacited by 42
  25. 2020Privacy Amplification via Random Check-Inscited by 39
  26. 2018Scattering Networks for Hybrid Representation Learningcited by 33
  27. 2020Evading Curse of Dimensionality in Unconstrained Private GLMs via Private Gradient Descentcited by 27
  28. 2017On Connecting Stochastic Gradient MCMC and Differential Privacycited by 16

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 2.

4 papers.

  1. 2025DC-SGD: Differentially Private SGD with Dynamic Clipping through Gradient Norm Distribution Estimationcited by 6
  2. 2023Little is Enough: Boosting Privacy by Sharing Only Hard Labels in Federated Semi-Supervised Learningcited by 5
  3. 2024Private Linear Regression with Differential Privacy and PAC Privacycited by 0
  4. 2025Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)cited by 0