mlsanityData to 7 Oct 2026

Unlocking High-Accuracy Differentially Private Image Classification through Scale

arXiv 2204.13650 ↗First posted 28 Apr 2022cs.LG cs.CR cs.CVcited by 218

Not on the map. Why?

Abstract (arXiv)

Differential Privacy (DP) provides a formal privacy guarantee preventing adversaries with access to a machine learning model from extracting information about individual training points. Differentially Private Stochastic Gradient Descent (DP-SGD), the most popular DP training method for deep learning, realizes this protection by injecting noise during training. However previous works have found that DP-SGD often leads to a significant degradation in performance on standard image classification benchmarks. Furthermore, some authors have postulated that DP-SGD inherently performs poorly on large models, since the norm of the noise required to preserve privacy is proportional to the model dimension. In contrast, we demonstrate that DP-SGD on over-parameterized models can perform significantly better than previously thought. Combining careful hyper-parameter tuning with simple techniques to ensure signal propagation and improve the convergence rate, we obtain a new SOTA without extra data on CIFAR-10 of 81.4% under (8, 10-5)-DP using a 40-layer Wide-ResNet, improving over the previous SOTA of 71.7%. When fine-tuning a pre-trained NFNet-F3, we achieve a remarkable 83.8% top-1 accuracy on ImageNet under (0.5, 8*10-7)-DP. Additionally, we also achieve 86.7% top-1 accuracy under (8, 8 · 10-7)-DP, which is just 4.3% below the current non-private SOTA for this task. We believe our results are a significant step towards closing the accuracy gap between private and non-private image classification.

Key facts. Cited by 218 papers in the map (data through 7 Oct 2026). Builds on 77 of its 117 references that are in the map. First posted on arXiv 28 Apr 2022.

Builds on

77 of 117 references.

References · 77

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2016Wide Residual Networkscited by 3,057
  12. 2016Deep Learning with Differential Privacycited by 2,682
38 more
  1. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  2. 2020Extracting Training Data from Large Language Modelscited by 2,144
  3. 2018Group Normalizationcited by 1,514
  4. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  6. 2021Scaling Vision Transformerscited by 993
  7. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  8. 2021Membership Inference Attacks From First Principlescited by 839
  9. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  10. 2021Going deeper with Image Transformerscited by 759
  11. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  12. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  13. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
  14. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  15. 2020Label-Only Membership Inference Attackscited by 371
  16. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  17. 2019Differential Privacy Has Disparate Impact on Model Accuracycited by 316
  18. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  19. 2018An Empirical Model of Large-Batch Trainingcited by 268
  20. 2018Scalable Private Learning with PATEcited by 266
  21. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  22. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 233
  23. 2019Differentially Private Learning with Adaptive Clippingcited by 218
  24. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 213
  25. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 199
  26. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  27. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 193
  28. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  29. 2020Private Stochastic Convex Optimization: Optimal Rates in Linear Timecited by 152
  30. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  31. 2022Reconstructing Training Data with Informed Adversariescited by 143
  32. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 140
  33. 2020Tempered Sigmoid Activations for Deep Learning with Differential Privacycited by 121
  34. 2021Large-Scale Differentially Private BERTcited by 117
  35. 2022Revisiting Weakly Supervised Pre-Training of Visual Perception Modelscited by 116
  36. 2021Hyperparameter Tuning with Renyi Differential Privacycited by 104
  37. 2020Hiding Among the Clones: A Simple and Nearly Optimal Analysis of Privacy Amplification by Shufflingcited by 96
  38. 2020BYOL works even without batch statisticscited by 95

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 34; 2023: 65; 2024: 59; 2025: 37; 2026: 23.

218 papers.

Sort:

By area:

  1. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  2. 2023Privacy Auditing with One (1) Training Runcited by 119
  3. 2022Differentially Private Diffusion Modelscited by 118
  4. 2023Foundation Models and Fair Usecited by 112
  5. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
  6. 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 87
  7. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 86
  8. 2023Differentially Private Diffusion Models Generate Useful Synthetic Imagescited by 85
  9. 2022Differentially Private Optimization on Large Model at Small Costcited by 65
  10. 2023Federated Learning of Gboard Language Models with Differential Privacycited by 64
  11. 2023Differentially Private Synthetic Data via Foundation Model APIs 1: Imagescited by 63
  12. 2023DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineercited by 56
38 more
  1. 2022Exploring the Limits of Differentially Private Deep Learning with Group-wise Clippingcited by 52
  2. 2022TAN Without a Burn: Scaling Laws of DP-SGDcited by 51
  3. 2022When Does Differentially Private Learning Not Suffer in High Dimensions?cited by 48
  4. 2022Multi-Epoch Matrix Factorization Mechanisms for Private Machine Learningcited by 48
  5. 2022Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacycited by 47
  6. 2022Differentially Private Bias-Term Fine-tuning of Foundation Modelscited by 47
  7. 2023Bounding Training Data Reconstruction in DP-SGDcited by 42
  8. 2024Private Fine-tuning of Large Language Models with Zeroth-order Optimizationcited by 41
  9. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 40
  10. 2023Privacy Side Channels in Machine Learning Systemscited by 38
  11. 2023Why Is Public Pretraining Necessary for Private Model Training?cited by 36
  12. 2024Evaluations of Machine Learning Privacy Defenses are Misleadingcited by 35
  13. 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 34
  14. 2024Fast Exact Unlearning for In-Context Learning Data for LLMscited by 33
  15. 2023Elijah: Eliminating Backdoors Injected in Diffusion Models via Distribution Shiftcited by 31
  16. 2023DP-LDMs: Differentially Private Latent Diffusion Modelscited by 28
  17. 2024Fine-Tuning Large Language Models with User-Level Differential Privacycited by 28
  18. 2024Federated LoRA with Sparse Communicationcited by 26
  19. 2022No Free Lunch in "Privacy for Free: How does Dataset Condensation Help Privacy"cited by 23
  20. 2022Learning to Generate Image Embeddings with User-level Differential Privacycited by 23
  21. 2023Selective Pre-training for Private Fine-tuningcited by 22
  22. 2024How Private are DP-SGD Implementations?cited by 22
  23. 2022Individual Privacy Accounting for Differentially Private Stochastic Gradient Descentcited by 20
  24. 2022Beyond Uniform Lipschitz Condition in Differentially Private Optimizationcited by 19
  25. 2023Differentially Private Image Classification by Learning Priors from Random Processescited by 19
  26. 2024Nearly Tight Black-Box Auditing of Differentially Private Machine Learningcited by 19
  27. 2022Differentially Private Adaptive Optimization with Delayed Preconditionerscited by 18
  28. 2023Practical Differentially Private and Byzantine-resilient Federated Learningcited by 18
  29. 2024Differentially Private Fine-Tuning of Diffusion Modelscited by 18
  30. 2023Private, fair and accurate: Training large-scale, privacy-preserving AI models in medical imagingcited by 17
  31. 2023Private GANs, Revisitedcited by 17
  32. 2023Exploring the Benefits of Visual Prompting in Differential Privacycited by 17
  33. 2023ViP: A Differentially Private Foundation Model for Computer Visioncited by 17
  34. 2023Privacy-Aware Document Visual Question Answeringcited by 17
  35. 2024Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Modelcited by 17
  36. 2023Unlocking Accuracy and Fairness in Differentially Private Image Classificationcited by 16
  37. 2023Rethinking Privacy in Machine Learning Pipelines from an Information Flow Control Perspectivecited by 16
  38. 2024Scalable DP-SGD: Shuffling vs. Poisson Subsamplingcited by 16

50 most cited of 218 All 218 (JSON)