mlsanityData to 7 Oct 2026

Private Distribution Learning with Public Data: The View from Sample Compression

arXiv 2308.06239 ↗First posted 11 Aug 2023cs.LG cs.CR stat.MLcited by 14

Not on the map. Why?

Abstract (arXiv)

We study the problem of private distribution learning with access to public data. In this setup, which we refer to as public-private learning, the learner is given public and private samples drawn from an unknown distribution p belonging to a class mathcal Q, with the goal of outputting an estimate of p while adhering to privacy constraints (here, pure differential privacy) only with respect to the private samples. We show that the public-private learnability of a class mathcal Q is connected to the existence of a sample compression scheme for mathcal Q, as well as to an intermediate notion we refer to as list learning. Leveraging this connection: (1) approximately recovers previous results on Gaussians over mathbb Rd; and (2) leads to new ones, including sample complexity upper bounds for arbitrary k-mixtures of Gaussians over mathbb Rd, results for agnostic and distribution-shift resistant learners, as well as closure properties for public-private learnability under taking mixtures and products of distributions. Finally, via the connection to list learning, we show that for Gaussians in mathbb Rd, at least d public samples are necessary for private learnability, which is close to the known upper bound of d+1 public samples.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 48 of its 81 references that are in the map. First posted on arXiv 11 Aug 2023.

Builds on

48 of 81 references.

References · 48

  1. 2016Deep Learning with Differential Privacycited by 2,682
  2. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  3. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  4. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 347
  5. 2016Concentrated Differential Privacy: Simplifications, Extensions, and Lower Boundscited by 339
  6. 2018Scalable Private Learning with PATEcited by 266
  7. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 233
  8. 2017Finite Sample Differentially Private Confidence Intervalscited by 103
  9. 2018Privately Learning High-Dimensional Distributionscited by 103
  10. 2016Learning from Untrusted Datacited by 90
  11. 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 88
  12. 2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
36 more
  1. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 86
  2. 2017List-Decodable Robust Mean Estimation and Learning Mixtures of Spherical Gaussianscited by 70
  3. 2021Differential privacy and robust statistics in high dimensionscited by 64
  4. 2023Federated Learning of Gboard Language Models with Differential Privacycited by 64
  5. 2014Private Learning and Sanitization: Pure vs. Approximate Differential Privacycited by 62
  6. 2019Private Hypothesis Selectioncited by 60
  7. 2020Differentially Private Assouad, Fano, and Le Camcited by 54
  8. 2019Diffprivlib: The IBM Differential Privacy Librarycited by 51
  9. 2021Public Data-Assisted Mirror Descent for Private Model Trainingcited by 49
  10. 2022Robustness Implies Privacy in Statistical Estimationcited by 47
  11. 2015Sample compression schemes for VC classescited by 46
  12. 2020On the Sample Complexity of Privately Learning Unbounded High-Dimensional Gaussianscited by 41
  13. 2021Leveraging Public Data for Practical Private Query Releasecited by 39
  14. 2021Private and polynomial time algorithms for learning Gaussians and beyondcited by 39
  15. 2019Limits of Private Learning with Access to Public Datacited by 38
  16. 2021Private Robust Estimation by Stabilizing Convex Relaxationscited by 38
  17. 2019List Decodable Learning via Sum of Squarescited by 37
  18. 2023Why Is Public Pretraining Necessary for Private Model Training?cited by 36
  19. 2021FriendlyCore: Practical Differentially Private Aggregationcited by 34
  20. 2021A Private and Computationally-Efficient Estimator for Unbounded Gaussianscited by 34
  21. 2022Private Estimation with Public Datacited by 31
  22. 2019Differentially Private Algorithms for Learning Mixtures of Separated Gaussianscited by 27
  23. 2020Private Query Release Assisted by Public Datacited by 27
  24. 2017Better Agnostic Clustering Via Relaxed Tensor Normscited by 21
  25. 2023Private estimation algorithms for stochastic block models and mixture modelscited by 21
  26. 2020Learning from Mixtures of Private and Public Populationscited by 19
  27. 2023Polynomial Time and Private Learning of Unbounded Gaussian Mixture Modelscited by 19
  28. 2019Privately Answering Classification Queries in the Agnostic PAC Modelcited by 18
  29. 2021Differentially-Private Clustering of Easy Instancescited by 17
  30. 2017BLENDER: Enabling Local Search with a Hybrid Differential Privacy Modelcited by 14
  31. 2023Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distancecited by 14
  32. 2021Privately Learning Mixtures of Axis-Aligned Gaussianscited by 13
  33. 2018The Power of The Hybrid Model for Mean Estimationcited by 11
  34. 2022Learning-Augmented Private Algorithms for Multiple Quantile Releasecited by 5
  35. 2018Some techniques in density estimationcited by 4
  36. 2022Transfer Learning In Differential Privacy's Hybrid-Modelcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 8; 2025: 4.

14 papers.

Sort:

By area:

  1. 2025Stochastic Forward-Backward Deconvolution: Training Diffusion Models with Finite Noisy Datasetscited by 11
  2. 2023Mixtures of Gaussians are Privately Learnable with a Polynomial Number of Samplescited by 5
  3. 2024Oracle-Efficient Differentially Private Learning with Public Datacited by 5
  4. 2024Public-data Assisted Private Stochastic Optimization: Power and Limitationscited by 5
  5. 2024Sample Compression Unleashed: New Generalization Bounds for Real Valued Lossescited by 4
  6. 2023Sample-Optimal Locally Private Hypothesis Selection and the Provable Benefits of Interactivitycited by 3
  7. 2024Credit Attribution and Stable Compressioncited by 3
  8. 2024Locally Private Sampling with Public Datacited by 3
  9. 2024Differentially Private Prototypes for Imbalanced Transfer Learningcited by 2
  10. 2024Agnostic Private Density Estimation for GMMs via List Global Stabilitycited by 2
  11. 2025Optimal Differentially Private Sampling of Unbounded Gaussianscited by 1
  12. 2025The Parameterized Complexity of Computing the VC-Dimensioncited by 1
2 more
  1. 2024Sample-Efficient Private Learning of Mixtures of Gaussianscited by 0
  2. 2025Robust Learnability of Sample-Compressible Distributions under Noisy or Adversarial Perturbationscited by 0