mlsanityData to 14 Oct 2026

The Role of Adaptive Optimizers for Honest Private Hyperparameter Selection

arXiv 2111.04906 ↗First posted 9 Nov 2021stat.ML cs.CR cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

Hyperparameter optimization is a ubiquitous challenge in machine learning, and the performance of a trained model depends crucially upon their effective selection. While a rich set of tools exist for this purpose, there are currently no practical hyperparameter selection methods under the constraint of differential privacy (DP). We study honest hyperparameter selection for differentially private machine learning, in which the process of hyperparameter tuning is accounted for in the overall privacy budget. To this end, we i) show that standard composition tools outperform more advanced techniques in many settings, ii) empirically and theoretically demonstrate an intrinsic connection between the learning rate and clipping norm hyperparameters, iii) show that adaptive optimizers like DPAdam enjoy a significant advantage in the process of honest hyperparameter tuning, and iv) draw upon novel limiting behaviour of Adam in the DP setting to design a new and more efficient optimizer.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 21 of its 40 references that are in the map. First posted on arXiv 9 Nov 2021.

Builds on

21 of 40 references.

References · 21

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Deep Learning with Differential Privacycited by 2,728
  3. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  4. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  5. 2019AutoML: A Survey of the State-of-the-Artcited by 390
  6. 2013The Composition Theorem for Differential Privacycited by 297
  7. 2018Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergencescited by 242
  8. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 235
  9. 2019Differentially Private Learning with Adaptive Clippingcited by 220
  10. 2021Numerical Composition of Differential Privacycited by 180
  11. 2017Machine Learning Models that Remember Too Muchcited by 169
  12. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 126
9 more
  1. 2014Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Boundscited by 107
  2. 2021Hyperparameter Tuning with Renyi Differential Privacycited by 104
  3. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 88
  4. 2018A General Approach to Adding Differential Privacy to Iterative Training Procedurescited by 84
  5. 2018Private Selection from Private Candidatescited by 75
  6. 2019Differentially Private Model Publishing for Deep Learningcited by 71
  7. 2020Training Production Language Models without Memorizing User Datacited by 68
  8. 2019Automatic Discovery of Privacy-Utility Pareto Frontscited by 16
  9. 2015Differentially Private Bayesian Optimizationcited by 12

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 10; 2024: 5; 2025: 4.

25 papers.

Sort:

By area:

  1. 2022Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimizationcited by 36
  2. 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 35
  3. 2023DP-AdamBC: Your DP-Adam Is Actually DP-SGD (Unless You Apply Bias Correction)cited by 28
  4. 2023Selective Pre-training for Private Fine-tuningcited by 23
  5. 2023Practical Differentially Private Hyperparameter Tuning with Subsamplingcited by 19
  6. 2023Private GANs, Revisitedcited by 17
  7. 2023Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distancecited by 16
  8. 2023Improving the Privacy and Practicality of Objective Perturbation for Differentially Private Linear Learnerscited by 12
  9. 2022A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimizationcited by 11
  10. 2025Towards hyperparameter-free optimization with differential privacycited by 7
  11. 2022Revisiting Hyperparameter Tuning with Differential Privacycited by 6
  12. 2023Improved Differentially Private Regression via Gradient Boostingcited by 6
13 more
  1. 2025DC-SGD: Differentially Private SGD with Dynamic Clipping through Gradient Norm Distribution Estimationcited by 6
  2. 2024Revisiting Differentially Private Hyper-parameter Tuningcited by 5
  3. 2024Differentially Private Post-Processing for Fair Regressioncited by 5
  4. 2024Delving into Differentially Private Transformercited by 5
  5. 2022Self-Supervised Pretraining for Differentially Private Learningcited by 4
  6. 2023DP-HyPO: An Adaptive Private Hyperparameter Optimization Frameworkcited by 4
  7. 2022Privacy-Preserving Deep Learning Model for Covid-19 Disease Detectioncited by 3
  8. 2022DPVIm: Differentially Private Variational Inference Improvedcited by 2
  9. 2023Online Sensitivity Optimization in Differentially Private Learningcited by 2
  10. 2024Privacy Profiles for Private Selectioncited by 2
  11. 2023A Theoretical Analysis of Efficiency Constrained Utility-Privacy Bi-Objective Optimization in Federated Learningcited by 1
  12. 2025Differentially Private Hyperparameter Tuning using Local Bayesian Optimizationcited by 1
  13. 2025Accelerating Differentially Private Federated Learning via Adaptive Extrapolationcited by 1