mlsanityData to 7 Oct 2026

Differentially Private Learning Needs Better Features (or Much More Data)

arXiv 2011.11660 ↗First posted 23 Nov 2020cs.LG cs.CR stat.MLcited by 233

Not on the map. Why?

Abstract (arXiv)

We demonstrate that differentially private machine learning has not yet reached its "AlexNet moment" on many canonical vision tasks: linear models trained on handcrafted features significantly outperform end-to-end deep neural networks for moderate privacy budgets. To exceed the performance of handcrafted features, we show that private learning requires either much more private data, or access to features learned on public data from a similar domain. Our work introduces simple yet strong baselines for differentially private learning that can inform the evaluation of future progress in this area.

Key facts. Cited by 233 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 30; 2022: 57; 2023: 57; 2024: 38; 2025: 32; 2026: 19.

233 papers.

Sort:

By area:

  1. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  2. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  3. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  4. 2021Evaluating Gradient Inversion Attacks and Defenses in Federated Learningcited by 196
  5. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 169
  6. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  7. 2022The Privacy Onion Effect: Memorization is Relativecited by 128
  8. 2021The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregationcited by 127
  9. 2021When the Curious Abandon Honesty: Federated Learning Is Not Privatecited by 126
  10. 2022Differentially Private Diffusion Modelscited by 118
  11. 2021Deep Learning with Label Differential Privacycited by 113
  12. 2023On Provable Copyright Protection for Generative Modelscited by 97
38 more
  1. 2021Trustworthy AI: A Computational Perspectivecited by 93
  2. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 87
  3. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  4. 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 87
  5. 2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
  6. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 86
  7. 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 80
  8. 2021The Skellam Mechanism for Differentially Private Federated Learningcited by 66
  9. 2022Differentially Private Optimization on Large Model at Small Costcited by 65
  10. 2021Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecturecited by 63
  11. 2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learningcited by 54
  12. 2022Mixed Differential Privacy in Computer Visioncited by 53
  13. 2022Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learningcited by 52
  14. 2022Exploring the Limits of Differentially Private Deep Learning with Group-wise Clippingcited by 52
  15. 2022TAN Without a Burn: Scaling Laws of DP-SGDcited by 51
  16. 2021Public Data-Assisted Mirror Descent for Private Model Trainingcited by 49
  17. 2024UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Modelscited by 49
  18. 2022Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacycited by 47
  19. 2022Differentially Private Bias-Term Fine-tuning of Foundation Modelscited by 47
  20. 2022Differentially Private Learning Needs Hidden State (Or Much Faster Convergence)cited by 45
  21. 2022Bounding Training Data Reconstruction in Private (Deep) Learningcited by 43
  22. 2023Bounding Training Data Reconstruction in DP-SGDcited by 42
  23. 2024Private Fine-tuning of Large Language Models with Zeroth-order Optimizationcited by 41
  24. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 40
  25. 2023Why Is Public Pretraining Necessary for Private Model Training?cited by 36
  26. 2022Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimizationcited by 35
  27. 2024Evaluations of Machine Learning Privacy Defenses are Misleadingcited by 35
  28. 2022Private Set Generation with Discriminative Informationcited by 34
  29. 2022Just Fine-tune Twice: Selective Differential Privacy for Large Language Modelscited by 32
  30. 2022Submix: Practical Private Prediction for Large-Scale Language Modelscited by 31
  31. 2022Private Estimation with Public Datacited by 31
  32. 2021Privacy Regularization: Joint Privacy-Utility Optimization in Language Modelscited by 29
  33. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  34. 2021Benchmarking Differential Privacy and Federated Learning for BERT Modelscited by 28
  35. 2022Pre-trained Perceptual Features Improve Differentially Private Image Generationcited by 28
  36. 2022Differentially private training of residual networks with scale normalisationcited by 26
  37. 2021The Role of Adaptive Optimizers for Honest Private Hyperparameter Selectioncited by 25
  38. 2022Differentially Private Label Protection in Split Learningcited by 25

50 most cited of 233 All 233 (JSON)