# Efficient Hyperparameter Optimization for Differentially Private Deep Learning

arXiv: https://arxiv.org/abs/2108.03888 · HTML: https://mlsanity.com/paper/2108.03888/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 21 references that are in the map.
- First posted on arXiv 9 Aug 2021.

## Builds on (most cited first, 12 of 12)

- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Differentially Private Learning Needs Better Features (or Much More Data)](https://mlsanity.com/paper/2011.11660/) (2020)
- [Differentially Private Learning with Adaptive Clipping](https://mlsanity.com/paper/1905.03871/) (2019)
- [Tempered Sigmoid Activations for Deep Learning with Differential Privacy](https://mlsanity.com/paper/2007.14191/) (2020)
- [AdaCliP: Adaptive Clipping for Private SGD](https://mlsanity.com/paper/1908.07643/) (2019)
- [Concentrated Differentially Private Gradient Descent with Adaptive per-Iteration Privacy Budget](https://mlsanity.com/paper/1808.09501/) (2018)
- [Three Tools for Practical Differential Privacy](https://mlsanity.com/paper/1812.02890/) (2018)
- [Dopamine: Differentially Private Federated Learning on Medical Data](https://mlsanity.com/paper/2101.11693/) (2021)
- [Automatic Discovery of Privacy-Utility Pareto Fronts](https://mlsanity.com/paper/1905.10862/) (2019)
- [Differentially private federated deep learning for multi-site medical image segmentation](https://mlsanity.com/paper/2107.02586/) (2021)
- [Stochastic Adaptive Line Search for Differentially Private Optimization](https://mlsanity.com/paper/2008.07978/) (2020)
- [Differentially Private Model Selection with Penalized and Constrained Likelihood](https://mlsanity.com/paper/1607.04204/) (2016)

## Built on by (most cited first, 5 of 5)

- [Efficient Differentially Private Fine-Tuning of LLMs via Reinforcement Learning](https://mlsanity.com/paper/2507.22565/) (2025)
- [An Interactive Framework for Finding the Optimal Trade-off in Differential Privacy](https://mlsanity.com/paper/2509.04290/) (2025)
- [Exploring Machine Learning Privacy/Utility trade-off from a hyperparameters Lens](https://mlsanity.com/paper/2303.01819/) (2023)
- [Federated Computing -- Survey on Building Blocks, Extensions and Systems](https://mlsanity.com/paper/2404.02779/) (2024)
- [Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)](https://mlsanity.com/paper/2508.15141/) (2025)

## Abstract (arXiv)

Tuning the hyperparameters in the differentially private stochastic gradient descent (DPSGD) is a fundamental challenge. Unlike the typical SGD, private datasets cannot be used many times for hyperparameter search in DPSGD; e.g., via a grid search. Therefore, there is an essential need for algorithms that, within a given search space, can find near-optimal hyperparameters for the best achievable privacy-utility tradeoffs efficiently. We formulate this problem into a general optimization framework for establishing a desirable privacy-utility tradeoff, and systematically study three cost-effective algorithms for being used in the proposed framework: evolutionary, Bayesian, and reinforcement learning. Our experiments, for hyperparameter tuning in DPSGD conducted on MNIST and CIFAR-10 datasets, show that these three algorithms significantly outperform the widely used grid search baseline. As this paper offers a first-of-a-kind framework for hyperparameter tuning in DPSGD, we discuss existing challenges and open directions for future studies. As we believe our work has implications to be utilized in the pipeline of private deep learning, we open-source our code at https://github.com/AmanPriyanshu/DP-HyperparamTuning.