mlsanityData to 14 Oct 2026

Lower Bounds for Public-Private Learning under Distribution Shift

arXiv 2507.17895 ↗First posted 23 Jul 2025cs.LG cs.CRcited by 0

Not on the map. Why?

Abstract (arXiv)

The most effective differentially private machine learning algorithms in practice rely on an additional source of purportedly public data. This paradigm is most interesting when the two sources combine to be more than the sum of their parts. However, there are settings such as mean estimation where we have strong lower bounds, showing that when the two data sources have the same distribution, there is no complementary value to combining the two data sources. In this work we extend the known lower bounds for public-private learning to setting where the two data sources exhibit significant distribution shift. Our results apply to both Gaussian mean estimation where the two distributions have different means, and to Gaussian linear regression where the two distributions exhibit parameter shift. We find that when the shift is small (relative to the desired accuracy), either public or private data must be sufficiently abundant to estimate the private parameter. Conversely, when the shift is large, public data provides no benefit.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 41 of its 69 references that are in the map. First posted on arXiv 23 Jul 2025.

Builds on

41 of 69 references.

References · 41

  1. 2016Deep Learning with Differential Privacycited by 2,728
  2. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  3. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  4. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 235
  5. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
  6. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 218
  7. 2018Privately Learning High-Dimensional Distributionscited by 104
  8. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 88
  9. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 88
  10. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  11. 2013Fingerprinting Codes and the Price of Approximate Differential Privacycited by 82
  12. 2022Differentially Private Optimization on Large Model at Small Costcited by 65
29 more
  1. 2015Between Pure and Approximate Differential Privacycited by 56
  2. 2022Mixed Differential Privacy in Computer Visioncited by 54
  3. 2022Exploring the Limits of Differentially Private Deep Learning with Group-wise Clippingcited by 52
  4. 2021Public Data-Assisted Mirror Descent for Private Model Trainingcited by 49
  5. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 41
  6. 2021Leveraging Public Data for Practical Private Query Releasecited by 39
  7. 2023Why Is Public Pretraining Necessary for Private Model Training?cited by 37
  8. 2014Interactive Fingerprinting Codes and the Hardness of Preventing False Discoverycited by 35
  9. 2014Preventing False Discovery in Interactive Data Analysis is Hardcited by 34
  10. 2017Tight Lower Bounds for Differentially Private Selectioncited by 34
  11. 2022New Lower Bounds for Private Estimation and a Generalized Fingerprinting Lemmacited by 33
  12. 2022Private Estimation with Public Datacited by 32
  13. 2022Submix: Practical Private Prediction for Large-Scale Language Modelscited by 31
  14. 2020Private Query Release Assisted by Public Datacited by 27
  15. 2021Tight and Robust Private Mean Estimation with Few Userscited by 26
  16. 2023Score Attack: A Lower Bound Technique for Optimal Differentially Private Learningcited by 21
  17. 2023Better and Simpler Lower Bounds for Differentially Private Statistical Estimationcited by 17
  18. 2018The Limits of Post-Selection Generalizationcited by 16
  19. 2024Information Complexity of Stochastic Convex Optimization: Applications to Generalization and Memorizationcited by 14
  20. 2018The Power of The Hybrid Model for Mean Estimationcited by 11
  21. 2022Can Foundation Models Help Us Achieve Perfect Secrecy?cited by 9
  22. 2023Optimal Differentially Private Model Training with Public Datacited by 9
  23. 2024Lower Bounds for Private Estimation of Gaussian Covariance Matrices under All Reasonable Parameter Regimescited by 9
  24. 2023Smooth Lower Bounds for Differentially Private Algorithms via Padding-and-Permuting Fingerprinting Codescited by 8
  25. 2024Joint Selection: Adaptively Incorporating Public Information for Private Synthetic Datacited by 8
  26. 2024Public-data Assisted Private Stochastic Optimization: Power and Limitationscited by 6
  27. 2024Characterizing the Training Dynamics of Private Fine-tuning with Langevin diffusioncited by 4
  28. 2024Fingerprinting Codes Meet Geometry: Improved Lower Bounds for Private Query Release and Adaptive Data Analysiscited by 3
  29. 2024Privacy in Metalearning and Multitask Learning: Modeling and Separationscited by 1

Built on by

Not cited yet.