mlsanityData to 14 Oct 2026

Hyperparameter Tuning with Renyi Differential Privacy

arXiv 2110.03620 ↗First posted 7 Oct 2021cs.LG cs.CR cs.DScited by 104

Not on the map. Why?

Abstract (arXiv)

For many differentially private algorithms, such as the prominent noisy stochastic gradient descent (DP-SGD), the analysis needed to bound the privacy leakage of a single training run is well understood. However, few studies have reasoned about the privacy leakage resulting from the multiple training runs needed to fine tune the value of the training algorithm's hyperparameters. In this work, we first illustrate how simply setting hyperparameters based on non-private training runs can leak private information. Motivated by this observation, we then provide privacy guarantees for hyperparameter search procedures within the framework of Renyi Differential Privacy. Our results improve and extend the work of Liu and Talwar (STOC 2019). Our analysis supports our previous observation that tuning hyperparameters does indeed leak private information, but we prove that, under certain assumptions, this leakage is modest, as long as each candidate training run needed to select hyperparameters is itself differentially private.

Key facts. Cited by 104 papers in the map (data through 14 Oct 2026). Builds on 15 of its 33 references that are in the map. First posted on arXiv 7 Oct 2021.

Builds on

15 of 33 references.

References · 15

  1. 2016Deep Learning with Differential Privacycited by 2,728
  2. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  3. 2020Extracting Training Data from Large Language Modelscited by 2,189
  4. 2017Renyi Differential Privacycited by 650
  5. 2016Concentrated Differential Privacy: Simplifications, Extensions, and Lower Boundscited by 346
  6. 2018Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergencescited by 242
  7. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 204
  8. 2016Concentrated Differential Privacycited by 144
  9. 2020The Discrete Gaussian for Differential Privacycited by 135
  10. 2020Tempered Sigmoid Activations for Deep Learning with Differential Privacycited by 122
  11. 2014Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Boundscited by 107
  12. 2015Algorithmic Stability for Adaptive Data Analysiscited by 88
3 more
  1. 2013Fingerprinting Codes and the Price of Approximate Differential Privacycited by 82
  2. 2018Private Selection from Private Candidatescited by 75
  3. 2017Tight Lower Bounds for Differentially Private Selectioncited by 34

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 26; 2023: 26; 2024: 19; 2025: 23; 2026: 7.

104 papers.

Sort:

By area:

  1. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  2. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
  3. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 153
  4. 2022Differentially Private Diffusion Modelscited by 119
  5. 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 88
  6. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  7. 2023Differentially Private Diffusion Models Generate Useful Synthetic Imagescited by 86
  8. 2022FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learningcited by 66
  9. 2023Privacy-Preserving In-Context Learning for Large Language Modelscited by 50
  10. 2022Algorithms that Approximate Data Removal: New Results and Limitationscited by 40
  11. 2022On Privacy and Personalization in Cross-Silo Federated Learningcited by 36
  12. 2022Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimizationcited by 36
38 more
  1. 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 35
  2. 2022Private Non-Convex Federated Learning Without a Trusted Servercited by 26
  3. 2021The Role of Adaptive Optimizers for Honest Private Hyperparameter Selectioncited by 25
  4. 2022Learning to Generate Image Embeddings with User-level Differential Privacycited by 23
  5. 2023Selective Pre-training for Private Fine-tuningcited by 23
  6. 2022Fully Adaptive Composition in Differential Privacycited by 22
  7. 2022Differentially Private Learning with Per-Sample Adaptive Clippingcited by 21
  8. 2022Beyond Uniform Lipschitz Condition in Differentially Private Optimizationcited by 19
  9. 2022Easy Differentially Private Linear Regressioncited by 19
  10. 2023Practical Differentially Private Hyperparameter Tuning with Subsamplingcited by 19
  11. 2023Arbitrary Decisions are a Hidden Cost of Differentially Private Trainingcited by 19
  12. 2023Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distancecited by 16
  13. 2022Fine-Tuning with Differential Privacy Necessitates an Additional Hyperparameter Searchcited by 15
  14. 2022DPIS: An Enhanced Mechanism for Differentially Private SGD with Importance Samplingcited by 13
  15. 2023DPMLBench: Holistic Evaluation of Differentially Private Machine Learningcited by 13
  16. 2023Faster Differentially Private Convex Optimization via Second-Order Methodscited by 13
  17. 2023Privately Customizing Prefinetuning to Better Match User Data in Federated Learningcited by 12
  18. 2023Improving the Privacy and Practicality of Objective Perturbation for Differentially Private Linear Learnerscited by 12
  19. 2024Banded Square Root Matrix Factorization for Differentially Private Model Trainingcited by 12
  20. 2022A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimizationcited by 11
  21. 2023DP-SGD Without Clipping: The Lipschitz Neural Network Waycited by 11
  22. 2025InvisibleInk: High-Utility and Low-Cost Text Generation with Differential Privacycited by 10
  23. 2023An Empirical Analysis of Fairness Notions under Differential Privacycited by 9
  24. 2022Skellam Mixture Mechanism: a Novel Approach to Federated Learning with Differential Privacycited by 7
  25. 2022On Noisy Evaluation in Federated Hyperparameter Tuningcited by 7
  26. 2025Towards hyperparameter-free optimization with differential privacycited by 7
  27. 2022(Nearly) Optimal Private Linear Regression via Adaptive Clippingcited by 6
  28. 2022Revisiting Hyperparameter Tuning with Differential Privacycited by 6
  29. 2022Generalized PTR: User-Friendly Recipes for Data-Adaptive Algorithms with Differential Privacycited by 6
  30. 2023Blink: Link Local Differential Privacy in Graph Neural Networks via Bayesian Estimationcited by 6
  31. 2023Optimal Unbiased Randomizers for Regression with Label Differential Privacycited by 6
  32. 2024Differential Privacy Under Class Imbalance: Methods and Empirical Insightscited by 6
  33. 2025DC-SGD: Differentially Private SGD with Dynamic Clipping through Gradient Norm Distribution Estimationcited by 6
  34. 2021Federated Learning with Heterogeneous Differential Privacycited by 5
  35. 2024Revisiting Differentially Private Hyper-parameter Tuningcited by 5
  36. 2024Differentially Private Post-Processing for Fair Regressioncited by 5
  37. 2024Delving into Differentially Private Transformercited by 5
  38. 2025The Importance of Being Discrete: Measuring the Impact of Discretization in End-to-End Differentially Private Synthetic Datacited by 5

50 most cited of 104 All 104 (JSON)