mlsanityData to 14 Oct 2026

Optimal Differentially Private Model Training with Public Data

arXiv 2306.15056 ↗First posted 26 Jun 2023cs.LG cs.CR math.OCcited by 9

Not on the map. Why?

Abstract (arXiv)

Differential privacy (DP) ensures that training a machine learning model does not leak private data. In practice, we may have access to auxiliary public data that is free of privacy concerns. In this work, we assume access to a given amount of public data and settle the following fundamental open questions: 1. What is the optimal (worst-case) error of a DP model trained over a private data set while having access to side public data? 2. How can we harness public data to improve DP model training in practice? We consider these questions in both the local and central models of pure and approximate DP. To answer the first question, we prove tight (up to log factors) lower and upper bounds that characterize the optimal error rates of three fundamental problems: mean estimation, empirical risk minimization, and stochastic convex optimization. We show that the optimal error rates can be attained (up to log factors) by either discarding private data and training a public model, or treating public data like it is private and using an optimal DP algorithm. To address the second question, we develop novel algorithms that are "even more optimal" (i.e. better constants) than the asymptotically optimal approaches described above. For local DP mean estimation, our algorithm is optimal including constants. Empirically, our algorithms show benefits over the state-of-the-art.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 26 of its 76 references that are in the map. First posted on arXiv 26 Jun 2023.

Builds on

26 of 76 references.

References · 26

  1. 2020Extracting Training Data from Large Language Modelscited by 2,189
  2. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  3. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 348
  4. 2018Scalable Private Learning with PATEcited by 269
  5. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
  6. 2017Collecting Telemetry Data Privatelycited by 149
  7. 2018Protection Against Reconstruction and Its Applications in Private Federated Learningcited by 130
  8. 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 90
  9. 2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
  10. 2020Differentially Private Assouad, Fano, and Le Camcited by 55
  11. 2021Public Data-Assisted Mirror Descent for Private Model Trainingcited by 49
  12. 2020Differentially Private Language Models Benefit from Public Pre-trainingcited by 47
14 more
  1. 2021Improved Rates for Differentially Private Stochastic Convex Optimization with Heavy-Tailed Datacited by 47
  2. 2020New Oracle-Efficient Algorithms for Private Synthetic Data Releasecited by 43
  3. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 41
  4. 2022Optimal Algorithms for Mean Estimation under Local Differential Privacycited by 36
  5. 2022Private Estimation with Public Datacited by 32
  6. 2022Private Non-Convex Federated Learning Without a Trusted Servercited by 26
  7. 2020Near Instance-Optimality in Differential Privacycited by 23
  8. 2022Stochastic Differentially Private and Fair Learningcited by 16
  9. 2023Algorithmically Effective Differentially Private Synthetic Datacited by 11
  10. 2022Individualized PATE: Differentially Private Machine Learning with Individual Privacy Guaranteescited by 10
  11. 2021Combining Public and Private Datacited by 9
  12. 2021Output Perturbation for Differentially Private Convex Optimization: Faster and More Generalcited by 7
  13. 2022Differentially Private Sampling from Rashomon Sets, and the Universality of Langevin Diffusion for Convex Optimizationcited by 3
  14. 2022Private Stochastic Optimization With Large Worst-Case Lipschitz Parametercited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 2; 2026: 3.

9 papers.

  1. 2024How to Make the Gradients Small Privately: Improved Rates for Differentially Private Non-Convex Optimizationcited by 10
  2. 2023Optimal Locally Private Nonparametric Classification with Public Datacited by 7
  3. 2024Public-data Assisted Private Stochastic Optimization: Power and Limitationscited by 6
  4. 2024Locally Private Sampling with Public Datacited by 3
  5. 2025Lower Bounds for Public-Private Learning under Distribution Shiftcited by 0
  6. 2025Privacy-Aware Time Series Synthesis via Public Knowledge Distillationcited by 0
  7. 2026Integrating Feature Correlation in Differential Privacy with Applications in DP-ERMcited by 0
  8. 2026Unlearning with Asymmetric Sources: Improved Unlearning-Utility Trade-off with Public Datacited by 0
  9. 2026Limits of Personalizing Differential Privacy Budgetscited by 0