mlsanityData to 14 Oct 2026

Self-Supervised Pretraining for Differentially Private Learning

arXiv 2206.07125 ↗First posted 14 Jun 2022cs.CV cs.CRcited by 4

Not on the map. Why?

Abstract (arXiv)

We demonstrate self-supervised pretraining (SSP) is a scalable solution to deep learning with differential privacy (DP) regardless of the size of available public datasets in image classification. When facing the lack of public datasets, we show the features generated by SSP on only one single image enable a private classifier to obtain much better utility than the non-learned handcrafted features under the same privacy budget. When a moderate or large size public dataset is available, the features produced by SSP greatly outperform the features trained with labels on various complex private datasets under the same private budget. We also compared multiple DP-enabled training frameworks to train a private classifier on the features generated by SSP. Finally, we report a non-trivial utility 25.3% of a private ImageNet-1K dataset when ε=3. Our source code can be found at urlhttps://github.com/UnchartedRLab/SSP.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 28 of its 40 references that are in the map. First posted on arXiv 14 Jun 2022.

Builds on

28 of 40 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Deep Learning with Differential Privacycited by 2,728
  3. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  4. 2016Matching Networks for One Shot Learningcited by 2,255
  5. 2020Extracting Training Data from Large Language Modelscited by 2,189
  6. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  7. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  8. 2017Renyi Differential Privacycited by 650
  9. 2017Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)cited by 549
  10. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 383
  11. 2018Scalable Private Learning with PATEcited by 269
  12. 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
16 more
  1. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 235
  2. 2017Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?cited by 221
  3. 2016Using Deep Learning for Image-Based Plant Disease Detectioncited by 219
  4. 2019Deep Learning with Gaussian Differential Privacycited by 114
  5. 2020What makes instance discrimination good for transfer learning?cited by 111
  6. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  7. 2019A critical analysis of self-supervision, or what we can learn from a single imagecited by 76
  8. 2015Privacy for Free: Posterior Sampling and Stochastic Gradient Monte Carlocited by 63
  9. 2021PASS: An ImageNet replacement for self-supervised pretraining without humanscited by 51
  10. 2014Deep Roto-Translation Scattering for Object Classificationcited by 50
  11. 2018Deep Learning for Detecting Cyberbullying Across Multiple Social Media Platformscited by 37
  12. 2018Scattering Networks for Hybrid Representation Learningcited by 33
  13. 2021The Role of Adaptive Optimizers for Honest Private Hyperparameter Selectioncited by 25
  14. 2020Differentially Private Deep Learning with Direct Feedback Alignmentcited by 7
  15. 2021Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliabilitycited by 6
  16. 2021Photonic Differential Privacy with Direct Feedback Alignmentcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 1; 2024: 0; 2025: 0; 2026: 1.

4 papers.

  1. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 88
  2. 2022Scaling Laws Beyond Backpropagationcited by 4
  3. 2026The pretraining domain outweighs the training objective in setting the privacy-utility trade-off of differentially private medical image analysiscited by 2
  4. 2023Benefit from public unlabeled data: A Frangi filtering-based pretraining network for 3D cerebrovascular segmentationcited by 0