mlsanityData to 14 Oct 2026

PILLAR: How to make semi-private learning more effective

arXiv 2306.03962 ↗First posted 6 Jun 2023cs.LG cs.AI cs.CRcited by 10

Not on the map. Why?

Abstract (arXiv)

In Semi-Supervised Semi-Private (SP) learning, the learner has access to both public unlabelled and private labelled data. We propose a computationally efficient algorithm that, under mild assumptions on the data, provably achieves significantly lower private labelled sample complexity and can be efficiently run on real-world datasets. For this purpose, we leverage the features extracted by networks pre-trained on public (labelled or unlabelled) data, whose distribution can significantly differ from the one on which SP learning is performed. To validate its empirical effectiveness, we propose a wide variety of experiments under tight privacy constraints (ε = 0.1) and with a focus on low-data regimes. In all of these settings, our algorithm exhibits significantly improved performance over available baselines that use similar amounts of public data.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 43 of its 85 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

43 of 85 references.

References · 43

  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  2. 2016Deep Learning with Differential Privacycited by 2,728
  3. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  4. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  5. 2021Membership Inference Attacks From First Principlescited by 878
  6. 2017Renyi Differential Privacycited by 650
  7. 2014OpenML: networked science in machine learningcited by 526
  8. 2018Rotation Equivariant CNNs for Digital Pathologycited by 422
  9. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  10. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 383
  11. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 348
  12. 2019Differential Privacy Has Disparate Impact on Model Accuracycited by 316
31 more
  1. 2018Scalable Private Learning with PATEcited by 269
  2. 2021Enhanced Membership Inference Attacks against Machine Learning Modelscited by 248
  3. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 235
  4. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  5. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
  6. 2018Do CIFAR-10 Classifiers Generalize to CIFAR-10?cited by 200
  7. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 199
  8. 2021Numerical Composition of Differential Privacycited by 180
  9. 2021Solo-learn: A Library of Self-supervised Methods for Visual Representation Learningcited by 116
  10. 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 90
  11. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 88
  12. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 88
  13. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  14. 2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
  15. 2014Private Learning and Sanitization: Pure vs. Approximate Differential Privacycited by 63
  16. 2021Public Data-Assisted Mirror Descent for Private Model Trainingcited by 49
  17. 2022When Does Differentially Private Learning Not Suffer in High Dimensions?cited by 48
  18. 2021Private Adaptive Gradient Methods for Convex Optimizationcited by 46
  19. 2019Limits of Private Learning with Access to Public Datacited by 39
  20. 2022Private Adaptive Optimization with Side Informationcited by 38
  21. 2014Sample Complexity Bounds on Differentially Private Learning via Communication Complexitycited by 31
  22. 2020Evading Curse of Dimensionality in Unconstrained Private GLMs via Private Gradient Descentcited by 27
  23. 2022DP-PCA: Statistically Optimal and Differentially Private PCAcited by 24
  24. 2023Have it your way: Individualized Privacy Assignment for DP-SGDcited by 21
  25. 2014Characterizing the Sample Complexity of Private Learnerscited by 19
  26. 2022How unfair is private learning ?cited by 18
  27. 2020On the Rademacher Complexity of Linear Hypothesis Setscited by 17
  28. 2023Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distancecited by 16
  29. 2020Efficient, Noise-Tolerant, and Private Learning via Boostingcited by 15
  30. 2023What Are the Chances? Explaining the Epsilon Parameter in Differential Privacycited by 14
  31. 2019Efficient Private Algorithms for Learning Large-Margin Halfspacescited by 13

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 5; 2025: 1.

10 papers.

  1. 2024Evaluations of Machine Learning Privacy Defenses are Misleadingcited by 36
  2. 2023Differentially Private Image Classification by Learning Priors from Random Processescited by 19
  3. 2023Unlocking Accuracy and Fairness in Differentially Private Image Classificationcited by 16
  4. 2024Neural Collapse Meets Differential Privacy: Curious Behaviors of NoisyGD with Near-perfect Representation Learningcited by 8
  5. 2023Safeguarding Data in Multimodal AI: A Differentially Private Approach to CLIP Trainingcited by 7
  6. 2024Public-data Assisted Private Stochastic Optimization: Power and Limitationscited by 6
  7. 2023On the Benefits of Public Representations for Private Transfer Learning under Distribution Shiftcited by 5
  8. 2024Credit Attribution and Stable Compressioncited by 3
  9. 2024Provable Privacy with Non-Private Pre-Processingcited by 2
  10. 2025Privacy-Aware Time Series Synthesis via Public Knowledge Distillationcited by 0