mlsanityData to 14 Oct 2026

Public Data-Assisted Mirror Descent for Private Model Training

arXiv 2112.00193 ↗First posted 1 Dec 2021cs.LG cs.CRcited by 49

Not on the map. Why?

Abstract (arXiv)

In this paper, we revisit the problem of using in-distribution public data to improve the privacy/utility trade-offs for differentially private (DP) model training. (Here, public data refers to auxiliary data sets that have no privacy concerns.) We design a natural variant of DP mirror descent, where the DP gradients of the private/sensitive data act as the linear term, and the loss generated by the public data as the mirror map. We show that, for linear regression with feature vectors drawn from a non-isotropic sub-Gaussian distribution, our algorithm, PDA-DPMD (a variant of mirror descent), provides population risk guarantees that are asymptotically better than the best known guarantees under DP (without having access to public data), when the number of public data samples (nsf pub) is sufficiently large. We further show that our algorithm has natural "noise stability" properties that control the variance due to noise added to ensure DP. We demonstrate the efficacy of our algorithm by showing privacy/utility trade-offs on four benchmark datasets (StackOverflow, WikiText-2, CIFAR-10, and EMNIST). We show that our algorithm not only significantly improves over traditional DP-SGD, which does not have access to public data, but to our knowledge is the first to improve over DP-SGD on models that have been pre-trained with public data.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Builds on 30 of its 50 references that are in the map. First posted on arXiv 1 Dec 2021.

Builds on

30 of 50 references.

References · 30

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2016Pointer Sentinel Mixture Modelscited by 3,096
  3. 2016Deep Learning with Differential Privacycited by 2,728
  4. 2019Introduction to Online Convex Optimizationcited by 803
  5. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  6. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  7. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 348
  8. 2018Scalable Private Learning with PATEcited by 269
  9. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 235
  10. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  11. 2019Private Stochastic Convex Optimization with Optimal Ratescited by 186
  12. 2020Stability of Stochastic Gradient Descent on Nonsmooth Convex Lossescited by 173
18 more
  1. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 170
  2. 2018Differentially Private Empirical Risk Minimization Revisited: Faster and More Generalcited by 128
  3. 2020Tempered Sigmoid Activations for Deep Learning with Differential Privacycited by 122
  4. 2018Privacy Amplification by Iterationcited by 100
  5. 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 90
  6. 2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
  7. 2020CoinPress: Practical Private Mean and Covariance Estimationcited by 74
  8. 2018Privacy-preserving Predictioncited by 48
  9. 2021Private Adaptive Gradient Methods for Convex Optimizationcited by 46
  10. 2019Limits of Private Learning with Access to Public Datacited by 39
  11. 2021Leveraging Public Data for Practical Private Query Releasecited by 39
  12. 2017Learning Differentially Private Recurrent Language Modelscited by 36
  13. 2020Private Query Release Assisted by Public Datacited by 27
  14. 2014Private Empirical Risk Minimization Beyond the Worst Case: The Effect of the Constraint Set Geometrycited by 24
  15. 2020Learning from Mixtures of Private and Public Populationscited by 19
  16. 2019Privately Answering Classification Queries in the Agnostic PAC Modelcited by 18
  17. 2017BLENDER: Enabling Local Search with a Hybrid Differential Privacy Modelcited by 14
  18. 2020Fast Dimension Independent Private AdaGrad on Publicly Estimated Subspacescited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 9; 2023: 18; 2024: 8; 2025: 10; 2026: 4.

49 papers.

Sort:

By area:

  1. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 153
  2. 2022When Does Differentially Private Learning Not Suffer in High Dimensions?cited by 48
  3. 2022Private Adaptive Optimization with Side Informationcited by 38
  4. 2023Why Is Public Pretraining Necessary for Private Model Training?cited by 37
  5. 2022Private Estimation with Public Datacited by 32
  6. 2023Can Public Large Language Models Help Private Cross-device Federated Learning?cited by 30
  7. 2023Selective Pre-training for Private Fine-tuningcited by 23
  8. 2022FLAIR: Federated Learning Annotated Image Repositorycited by 22
  9. 2023Differentially Private Natural Language Models: Recent Advances and Future Directionscited by 20
  10. 2023Differentially Private Image Classification by Learning Priors from Random Processescited by 19
  11. 2022Differentially Private Adaptive Optimization with Delayed Preconditionerscited by 18
  12. 2023Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distancecited by 16
37 more
  1. 2023Private Distribution Learning with Public Data: The View from Sample Compressioncited by 14
  2. 2022A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimizationcited by 11
  3. 2023PILLAR: How to make semi-private learning more effectivecited by 10
  4. 2024How to Make the Gradients Small Privately: Improved Rates for Differentially Private Non-Convex Optimizationcited by 10
  5. 2023Optimal Differentially Private Model Training with Public Datacited by 9
  6. 2022Mixed Federated Learning: Joint Decentralized and Centralized Learningcited by 8
  7. 2023Differentially Private Stochastic Convex Optimization in (Non)-Euclidean Space Revisitedcited by 8
  8. 2024Pre-training Differentially Private Models with Limited Public Datacited by 8
  9. 2023Mean Estimation Under Heterogeneous Privacy: Some Privacy Can Be Freecited by 6
  10. 2023Client-Level Differential Privacy via Adaptive Intermediary in Federated Medical Imagingcited by 6
  11. 2023Mean Estimation Under Heterogeneous Privacy Demandscited by 6
  12. 2023Private Learning with Public Featurescited by 6
  13. 2024Oracle-Efficient Differentially Private Learning with Public Datacited by 6
  14. 2024Public-data Assisted Private Stochastic Optimization: Power and Limitationscited by 6
  15. 2022Learning-Augmented Private Algorithms for Multiple Quantile Releasecited by 5
  16. 2023On the Benefits of Public Representations for Private Transfer Learning under Distribution Shiftcited by 5
  17. 2023Private Matrix Factorization with Public Item Featurescited by 3
  18. 2023Coupling public and private gradient provably helps optimizationcited by 3
  19. 2024Private Geometric Mediancited by 3
  20. 2024HRNet: Differentially Private Hierarchical and Multi-Resolution Network for Human Mobility Data Synthesizationcited by 2
  21. 2024Private Estimation when Data and Privacy Demands are Correlatedcited by 2
  22. 2024Training Large ASR Encoders with Differential Privacycited by 2
  23. 2025Managing Correlations in Data and Privacy Demandcited by 2
  24. 2025Private Zeroth-Order Optimization with Public Datacited by 2
  25. 2022Recycling Scraps: Improving Private Learning by Leveraging Intermediate Checkpointscited by 1
  26. 2025Mirror Descent Using the Tempesta Generalized Multi-parametric Logarithmscited by 1
  27. 2025A mean teacher algorithm for unlearning of language modelscited by 0
  28. 2025FedDuA: Doubly Adaptive Federated Learningcited by 0
  29. 2025Lower Bounds for Public-Private Learning under Distribution Shiftcited by 0
  30. 2025Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)cited by 0
  31. 2025Enhancing Differentially Private Linear Regression via Public Second-Momentcited by 0
  32. 2025Privacy-Aware Time Series Synthesis via Public Knowledge Distillationcited by 0
  33. 2025FusionDP: Foundation Model-Assisted Differentially Private Learning for Partially Sensitive Featurescited by 0
  34. 2026Differentially Private Truncation of Unbounded Data via Public Second Momentscited by 0
  35. 2026Deep Learning under Fractional-Order Differential Privacycited by 0
  36. 2026DP-KFC: Data-Free Preconditioning for Privacy-Preserving Deep Learningcited by 0
  37. 2026Escaping Iterative Parameter-Space Noise: Differentially Private Learning with a Hypernetworkcited by 0