mlsanityData to 7 Oct 2026

Why Is Public Pretraining Necessary for Private Model Training?

arXiv 2302.09483 ↗First posted 19 Feb 2023cs.LGcited by 36

Not on the map. Why?

Abstract (arXiv)

In the privacy-utility tradeoff of a model trained on benchmark language and vision tasks, remarkable improvements have been widely reported with the use of pretraining on publicly available data. This is in part due to the benefits of transfer learning, which is the standard motivation for pretraining in non-private settings. However, the stark contrast in the improvement achieved through pretraining under privacy compared to non-private settings suggests that there may be a deeper, distinct cause driving these gains. To explain this phenomenon, we hypothesize that the non-convex loss landscape of a model training necessitates an optimization algorithm to go through two phases. In the first, the algorithm needs to select a good "basin" in the loss landscape. In the second, the algorithm solves an easy optimization within that basin. The former is a harder problem to solve with private data, while the latter is harder to solve with public data due to a distribution shift or data scarcity. Guided by this intuition, we provide theoretical constructions that provably demonstrate the separation between private training with and without public pretraining. Further, systematic experiments on CIFAR10 and LibriSpeech provide supporting evidence for our hypothesis.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 39 of its 55 references that are in the map. First posted on arXiv 19 Feb 2023.

Builds on

39 of 55 references.

References · 39

  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  2. 2016Deep Learning with Differential Privacycited by 2,682
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  5. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  6. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  7. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  8. 2013The Composition Theorem for Differential Privacycited by 290
  9. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  10. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 233
  11. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  12. 2019Private Stochastic Convex Optimization with Optimal Ratescited by 183
27 more
  1. 2020Stability of Stochastic Gradient Descent on Nonsmooth Convex Lossescited by 172
  2. 2020Private Stochastic Convex Optimization: Optimal Rates in Linear Timecited by 152
  3. 2020DART: Open-Domain Structured Data Record to Text Generationcited by 144
  4. 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 88
  5. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  6. 2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
  7. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 86
  8. 2022Differentially Private Optimization on Large Model at Small Costcited by 65
  9. 2022Mixed Differential Privacy in Computer Visioncited by 53
  10. 2022Exploring the Limits of Differentially Private Deep Learning with Group-wise Clippingcited by 52
  11. 2021Public Data-Assisted Mirror Descent for Private Model Trainingcited by 49
  12. 2022When Does Differentially Private Learning Not Suffer in High Dimensions?cited by 48
  13. 2022Private Convex Optimization via Exponential Mechanismcited by 47
  14. 2022Differentially Private Bias-Term Fine-tuning of Foundation Modelscited by 47
  15. 2020Differentially Private Language Models Benefit from Public Pre-trainingcited by 46
  16. 2021Private Adaptive Gradient Methods for Convex Optimizationcited by 46
  17. 2021Leveraging Public Data for Practical Private Query Releasecited by 39
  18. 2019Limits of Private Learning with Access to Public Datacited by 38
  19. 2017Tight Lower Bounds for Differentially Private Selectioncited by 33
  20. 2018Concentrated Differentially Private Gradient Descent with Adaptive per-Iteration Privacy Budgetcited by 31
  21. 2022Submix: Practical Private Prediction for Large-Scale Language Modelscited by 31
  22. 2022Private Estimation with Public Datacited by 31
  23. 2020Private Query Release Assisted by Public Datacited by 27
  24. 2021Adapting to Function Difficulty and Growth Conditions in Private Optimizationcited by 19
  25. 2022DP2-VAE: Differentially Private Pre-trained Variational Autoencoderscited by 13
  26. 2018The Power of The Hybrid Model for Mean Estimationcited by 11
  27. 2021Dynamic Privacy Budget Allocation Improves Data Efficiency of Differentially Private Gradient Descentcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 11; 2025: 6; 2026: 4.

36 papers.

Sort:

By area:

  1. 2023Differentially Private Synthetic Data via Foundation Model APIs 1: Imagescited by 63
  2. 2024Private Fine-tuning of Large Language Models with Zeroth-order Optimizationcited by 41
  3. 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 34
  4. 2023Can Public Large Language Models Help Private Cross-device Federated Learning?cited by 30
  5. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  6. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  7. 2023Selective Pre-training for Private Fine-tuningcited by 22
  8. 2023Differentially Private Image Classification by Learning Priors from Random Processescited by 19
  9. 2023Unlocking Accuracy and Fairness in Differentially Private Image Classificationcited by 16
  10. 2023Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distancecited by 14
  11. 2023Private Distribution Learning with Public Data: The View from Sample Compressioncited by 14
  12. 2023TMI! Finetuned Models Leak Private Information from their Pretraining Datacited by 13
24 more
  1. 2023Private Learning with Public Featurescited by 6
  2. 2023Unintended Memorization in Large ASR Models, and How to Mitigate Itcited by 5
  3. 2023On the Benefits of Public Representations for Private Transfer Learning under Distribution Shiftcited by 5
  4. 2024Public-data Assisted Private Stochastic Optimization: Power and Limitationscited by 5
  5. 2023Differentially Private Non-Convex Optimization under the KL Condition with Optimal Ratescited by 4
  6. 2024Characterizing the Training Dynamics of Private Fine-tuning with Langevin diffusioncited by 4
  7. 2024Adaptively Private Next-Token Prediction of Large Language Modelscited by 4
  8. 2024Private Geometric Mediancited by 3
  9. 2024Provable Privacy with Non-Private Pre-Processingcited by 2
  10. 2024Differentially Private Prototypes for Imbalanced Transfer Learningcited by 2
  11. 2024Training Large ASR Encoders with Differential Privacycited by 2
  12. 2025An Optimization Framework for Differentially Private Sparse Fine-Tuningcited by 2
  13. 2025Leveraging Vertical Public-Private Split for Improved Synthetic Data Generationcited by 2
  14. 2025Machine Learning with Privacy for Protected Attributescited by 2
  15. 2023Population Expansion for Training Language Models with Private Federated Learningcited by 1
  16. 2023Inference and Interference: The Role of Clipping, Pruning and Loss Landscapes in Differentially Private Stochastic Gradient Descentcited by 1
  17. 2025PrAda-GAN: A Private Adaptive Generative Adversarial Network with Bayes Network Structurecited by 1
  18. 2024Mitigating Noise Detriment in Differentially Private Federated Learning with Model Pre-trainingcited by 0
  19. 2025Securing Transfer-Learned Networks with Reverse Homomorphic Encryptioncited by 0
  20. 2025Lower Bounds for Public-Private Learning under Distribution Shiftcited by 0
  21. 2026Differentially Private and Communication Efficient Large Language Model Split Inference via Stochastic Quantization and Soft Promptcited by 0
  22. 2026Unlearning with Asymmetric Sources: Improved Unlearning-Utility Trade-off with Public Datacited by 0
  23. 2026Token by Token, Compromised: Backdoor Vulnerabilities in Unified Autoregressive Modelscited by 0
  24. 2026StraightDP: Geometry-Aware Differential Privacy for Rectified-Flow Transformerscited by 0