mlsanityData to 7 Oct 2026

Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distance

arXiv 2303.01256 ↗First posted 2 Mar 2023stat.ML cs.CR cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

Differentially private stochastic gradient descent privatizes model training by injecting noise into each iteration, where the noise magnitude increases with the number of model parameters. Recent works suggest that we can reduce the noise by leveraging public data for private machine learning, by projecting gradients onto a subspace prescribed by the public data. However, given a choice of public datasets, it is not a priori clear which one may be most appropriate for the private task. We give an algorithm for selecting a public dataset by measuring a low-dimensional subspace distance between gradients of the public and private examples. We provide theoretical analysis demonstrating that the excess risk scales with this subspace distance. This distance is easy to compute and robust to modifications in the setting. Empirical evaluation shows that trained model accuracy is monotone in this distance.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 48 of its 104 references that are in the map. First posted on arXiv 2 Mar 2023.

Builds on

48 of 104 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2016Densely Connected Convolutional Networkscited by 6,691
  6. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  9. 2016Deep Learning with Differential Privacycited by 2,682
  10. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  11. 2020Extracting Training Data from Large Language Modelscited by 2,144
  12. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
36 more
  1. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  2. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
  3. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  4. 2018Deep Visual Domain Adaptation: A Surveycited by 634
  5. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  6. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  7. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  8. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  9. 2018Adversarial Robustness Toolbox v1.0.0cited by 214
  10. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  11. 2019Task2Vec: Task Embedding for Meta-Learningcited by 197
  12. 2020Geometric Dataset Distances via Optimal Transportcited by 157
  13. 2021Hyperparameter Tuning with Renyi Differential Privacycited by 104
  14. 2020Deep Visual Domain Adaptationcited by 91
  15. 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 88
  16. 2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
  17. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 86
  18. 2020CoinPress: Practical Private Mean and Covariance Estimationcited by 74
  19. 2019Private Hypothesis Selectioncited by 60
  20. 2020Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorizationcited by 58
  21. 2022Mixed Differential Privacy in Computer Visioncited by 53
  22. 2019Transfer Adaptation Learning: A Decade Surveycited by 52
  23. 2021Public Data-Assisted Mirror Descent for Private Model Trainingcited by 49
  24. 2022When Does Differentially Private Learning Not Suffer in High Dimensions?cited by 48
  25. 2021Improved Rates for Differentially Private Stochastic Convex Optimization with Heavy-Tailed Datacited by 47
  26. 2022Differentially Private Bias-Term Fine-tuning of Foundation Modelscited by 47
  27. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 40
  28. 2022Private Adaptive Optimization with Side Informationcited by 38
  29. 2019Hessian based analysis of SGD for Deep Nets: Dynamics and Generalizationcited by 37
  30. 2022Hidden Poison: Machine Unlearning Enables Camouflaged Poisoning Attackscited by 37
  31. 2023Why Is Public Pretraining Necessary for Private Model Training?cited by 36
  32. 2018Concentrated Differentially Private Gradient Descent with Adaptive per-Iteration Privacy Budgetcited by 31
  33. 2021The Role of Adaptive Optimizers for Honest Private Hyperparameter Selectioncited by 25
  34. 2022DP-PCA: Statistically Optimal and Differentially Private PCAcited by 24
  35. 2022Fine-Tuning with Differential Privacy Necessitates an Additional Hyperparameter Searchcited by 15
  36. 2020Fast Dimension Independent Private AdaGrad on Publicly Estimated Subspacescited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 4; 2025: 5; 2026: 1.

14 papers.

Sort:

By area:

  1. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  2. 2023Selective Pre-training for Private Fine-tuningcited by 22
  3. 2023Private Distribution Learning with Public Data: The View from Sample Compressioncited by 14
  4. 2023PILLAR: How to make semi-private learning more effectivecited by 10
  5. 2023Optimal Locally Private Nonparametric Classification with Public Datacited by 7
  6. 2024Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tailscited by 4
  7. 2024Differentially Private Prototypes for Imbalanced Transfer Learningcited by 2
  8. 2025Memory-Efficient Differentially Private Training with Gradient Random Projectioncited by 2
  9. 2025Correlating Cross-Iteration Noise for DP-SGD using Model Curvaturecited by 1
  10. 2025Cocoon: A System Architecture for Differentially Private Training with Correlated Noisescited by 1
  11. 2024Differentially Private Active Learning: Balancing Effective Data Selection and Privacycited by 0
  12. 2025Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)cited by 0
2 more
  1. 2025Balancing Utility and Privacy: Dynamically Private SGD with Random Projectioncited by 0
  2. 2026Curation Leaks: Membership Inference Attacks against Data Curation for Machine Learningcited by 0