# Dynamic Privacy Budget Allocation Improves Data Efficiency of Differentially Private Gradient Descent

arXiv: https://arxiv.org/abs/2101.07413 · HTML: https://mlsanity.com/paper/2101.07413/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 43 references that are in the map.
- First posted on arXiv 19 Jan 2021.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Condition](https://mlsanity.com/paper/1608.04636/) (2016)
- [Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds](https://mlsanity.com/paper/1605.02065/) (2016)
- [Differentially Private Learning with Adaptive Clipping](https://mlsanity.com/paper/1905.03871/) (2019)
- [Private Stochastic Convex Optimization with Optimal Rates](https://mlsanity.com/paper/1908.09970/) (2019)
- [Private Stochastic Convex Optimization: Optimal Rates in Linear Time](https://mlsanity.com/paper/2005.04763/) (2020)
- [Stochastic Variance Reduction for Nonconvex Optimization](https://mlsanity.com/paper/1603.06160/) (2016)
- [Differentially Private Empirical Risk Minimization Revisited: Faster and More General](https://mlsanity.com/paper/1802.05251/) (2018)
- [AdaCliP: Adaptive Clipping for Private SGD](https://mlsanity.com/paper/1908.07643/) (2019)
- [Differentially Private Model Publishing for Deep Learning](https://mlsanity.com/paper/1904.02200/) (2019)
- [SoK: Differential Privacies](https://mlsanity.com/paper/1906.01337/) (2019)
- [Concentrated Differentially Private Gradient Descent with Adaptive per-Iteration Privacy Budget](https://mlsanity.com/paper/1808.09501/) (2018)
- [Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds](https://mlsanity.com/paper/2006.13501/) (2020)
- [Dual Policy Distillation](https://mlsanity.com/paper/2006.04061/) (2020)
- [An Adaptive and Fast Convergent Approach to Differentially Private Deep Learning](https://mlsanity.com/paper/1912.09150/) (2019)
- [Differential Privacy for Growing Databases](https://mlsanity.com/paper/1803.06416/) (2018)

## Built on by (most cited first, 5 of 5)

- [Why Is Public Pretraining Necessary for Private Model Training?](https://mlsanity.com/paper/2302.09483/) (2023)
- [DOPPLER: Differentially Private Optimizers with Low-pass Filter for Privacy Noise Reduction](https://mlsanity.com/paper/2408.13460/) (2024)
- [Distributed Harmonization: Federated Clustered Batch Effect Adjustment and Generalization](https://mlsanity.com/paper/2405.15081/) (2024)
- [Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyond](https://mlsanity.com/paper/2410.18114/) (2024)
- [BGTplanner: Maximizing Training Accuracy for Differentially Private Federated Recommenders via Strategic Privacy Budget Allocation](https://mlsanity.com/paper/2412.02934/) (2024)

## Abstract (arXiv)

Protecting privacy in learning while maintaining the model performance has become increasingly critical in many applications that involve sensitive data. Private Gradient Descent (PGD) is a commonly used private learning framework, which noises gradients based on the Differential Privacy protocol. Recent studies show that dynamic privacy schedules of decreasing noise magnitudes can improve loss at the final iteration, and yet theoretical understandings of the effectiveness of such schedules and their connections to optimization algorithms remain limited. In this paper, we provide comprehensive analysis of noise influence in dynamic privacy schedules to answer these critical questions. We first present a dynamic noise schedule minimizing the utility upper bound of PGD, and show how the noise influence from each optimization step collectively impacts utility of the final model. Our study also reveals how impacts from dynamic noise influence change when momentum is used. We empirically show the connection exists for general non-convex losses, and the influence is greatly impacted by the loss curvature.