mlsanityData to 14 Oct 2026

Differentially Private Image Classification by Learning Priors from Random Processes

arXiv 2306.06076 ↗First posted 8 Jun 2023cs.CV cs.CR cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

In privacy-preserving machine learning, differentially private stochastic gradient descent (DP-SGD) performs worse than SGD due to per-sample gradient clipping and noise addition. A recent focus in private learning research is improving the performance of DP-SGD on private data by incorporating priors that are learned on real-world public data. In this work, we explore how we can improve the privacy-utility tradeoff of DP-SGD by learning priors from images generated by random processes and transferring these priors to private data. We propose DP-RandP, a three-phase approach. We attain new state-of-the-art accuracy when training from scratch on CIFAR10, CIFAR100, MedMNIST and ImageNet for a range of privacy budgets varepsilon ∈ [1, 8]. In particular, we improve the previous best reported accuracy on CIFAR10 from 60.6 % to 72.3 % for varepsilon=1.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 51 of its 74 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

51 of 74 references.

References · 51

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  7. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  8. 2016Deep Learning with Differential Privacycited by 2,728
  9. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  10. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  11. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  12. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
38 more
  1. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 690
  2. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  3. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  4. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 383
  5. 2018Rethinking ImageNet Pre-trainingcited by 377
  6. 2019Gauge Equivariant Convolutional Networks and the Icosahedral CNNcited by 267
  7. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 235
  8. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
  9. 2018Improving the Gaussian Mechanism for Differential Privacy: Analytical Calibration and Optimal Denoisingcited by 218
  10. 2020MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysiscited by 206
  11. 2021Numerical Composition of Differential Privacycited by 180
  12. 2022Reconstructing Training Data with Informed Adversariescited by 148
  13. 2020Tempered Sigmoid Activations for Deep Learning with Differential Privacycited by 122
  14. 2021Pre-training without Natural Imagescited by 100
  15. 2021Learning to See by Looking at Noisecited by 92
  16. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 88
  17. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 88
  18. 2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
  19. 2023Differentially Private Diffusion Models Generate Useful Synthetic Imagescited by 86
  20. 2022Head2Toe: Utilizing Intermediate Representations for Better Transfer Learningcited by 83
  21. 2022Differentially Private Optimization on Large Model at Small Costcited by 65
  22. 2022TAN Without a Burn: Scaling Laws of DP-SGDcited by 52
  23. 2022Exploring the Limits of Differentially Private Deep Learning with Group-wise Clippingcited by 52
  24. 2014Deep Roto-Translation Scattering for Object Classificationcited by 50
  25. 2021Public Data-Assisted Mirror Descent for Private Model Trainingcited by 49
  26. 2022When Does Differentially Private Learning Not Suffer in High Dimensions?cited by 48
  27. 2022Private Adaptive Optimization with Side Informationcited by 38
  28. 2019Augment your batch: better training with larger batchescited by 37
  29. 2023Why Is Public Pretraining Necessary for Private Model Training?cited by 37
  30. 2021On the Convergence and Calibration of Deep Learning with Differential Privacycited by 36
  31. 2021Fast Adaptation with Linearized Neural Networkscited by 32
  32. 2020Gradients as Features for Deep Representation Learningcited by 30
  33. 2022Procedural Image Programs for Representation Learningcited by 28
  34. 2022Differentially private training of residual networks with scale normalisationcited by 26
  35. 2021Not all noise is accounted equally: How differentially private learning benefits from large sampling ratescited by 21
  36. 2023A Bias-Accuracy-Privacy Trilemma for Statistical Estimationcited by 12
  37. 2023PILLAR: How to make semi-private learning more effectivecited by 10
  38. 2022Differentially Private Image Classification from Featurescited by 7

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 9; 2025: 5; 2026: 4.

19 papers.

Sort:

By area:

  1. 2024Private Fine-tuning of Large Language Models with Zeroth-order Optimizationcited by 41
  2. 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 35
  3. 2023Unlocking Accuracy and Fairness in Differentially Private Image Classificationcited by 16
  4. 2024Neural Collapse Meets Differential Privacy: Curious Behaviors of NoisyGD with Near-perfect Representation Learningcited by 8
  5. 2024Investigating the Quality of DermaMNIST and Fitzpatrick17k Dermatological Image Datasetscited by 7
  6. 2024Differentially Private Representation Learning via Image Captioningcited by 5
  7. 2024Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tailscited by 5
  8. 2024Characterizing the Training Dynamics of Private Fine-tuning with Langevin diffusioncited by 4
  9. 2025Dual-Priv Pruning : Efficient Differential Private Fine-Tuning in Multimodal Large Language Modelscited by 4
  10. 2025Understanding Private Learning From Feature Perspectivecited by 3
  11. 2024Noise-Aware Differentially Private Regression via Meta-Learningcited by 2
  12. 2024Bayesian Power Steering: An Effective Approach for Domain Adaptation of Diffusion Modelscited by 1
7 more
  1. 2026Private Prediction via PAC Privacycited by 1
  2. 2025Data Efficient Subset Training with Differential Privacycited by 0
  3. 2025Separating Knowledge and Perception with Procedural Datacited by 0
  4. 2025Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)cited by 0
  5. 2026Rethinking Benchmarks for Differentially Private Image Classificationcited by 0
  6. 2026Differentially Private Natural Gradient Descentcited by 0
  7. 2026End-to-End Differential Privacy in Training Deep Neural Network Classifierscited by 0