# Concentrated Differentially Private Gradient Descent with Adaptive per-Iteration Privacy Budget

arXiv: https://arxiv.org/abs/1808.09501 · HTML: https://mlsanity.com/paper/1808.09501/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Aug 2018.

## Built on by (most cited first, 30 of 32)

- [Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning](https://mlsanity.com/paper/2102.12677/) (2021)
- [Mixed Differential Privacy in Computer Vision](https://mlsanity.com/paper/2203.11481/) (2022)
- [Survey: Leakage and Privacy at Inference Time](https://mlsanity.com/paper/2107.01614/) (2021)
- [Why Is Public Pretraining Necessary for Private Model Training?](https://mlsanity.com/paper/2302.09483/) (2023)
- [Differentially private inference via noisy optimization](https://mlsanity.com/paper/2103.11003/) (2021)
- [DPNAS: Neural Architecture Search for Deep Learning with Differential Privacy](https://mlsanity.com/paper/2110.08557/) (2021)
- [Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distance](https://mlsanity.com/paper/2303.01256/) (2023)
- [DPIS: An Enhanced Mechanism for Differentially Private SGD with Importance Sampling](https://mlsanity.com/paper/2210.09634/) (2022)
- [Adap DP-FL: Differentially Private Federated Learning with Adaptive Noise](https://mlsanity.com/paper/2211.15893/) (2022)
- [DPlis: Boosting Utility of Differentially Private Deep Learning via Randomized Smoothing](https://mlsanity.com/paper/2103.01496/) (2021)
- [DPAR: Decoupled Graph Neural Networks with Node-Level Differential Privacy](https://mlsanity.com/paper/2210.04442/) (2022)
- [DPSUR: Accelerating Differentially Private Stochastic Gradient Descent Using Selective Update and Release](https://mlsanity.com/paper/2311.14056/) (2023)
- [Spectral-DP: Differentially Private Deep Learning through Spectral Perturbation and Filtering](https://mlsanity.com/paper/2307.13231/) (2023)
- [Layered Randomized Quantization for Communication-Efficient and Privacy-Preserving Distributed Learning](https://mlsanity.com/paper/2312.07060/) (2023)
- [Dynamic Privacy Budget Allocation Improves Data Efficiency of Differentially Private Gradient Descent](https://mlsanity.com/paper/2101.07413/) (2021)
- [Efficient Hyperparameter Optimization for Differentially Private Deep Learning](https://mlsanity.com/paper/2108.03888/) (2021)
- [Adaptive Differentially Private Empirical Risk Minimization](https://mlsanity.com/paper/2110.07435/) (2021)
- [Heterogeneous Decentralized Machine Unlearning with Seed Model Distillation](https://mlsanity.com/paper/2308.13269/) (2023)
- [Noise-Augmented Privacy-Preserving Empirical Risk Minimization with Dual-purpose Regularizer and Privacy Budget Retrieval and Recycling](https://mlsanity.com/paper/2110.08676/) (2021)
- [Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix](https://mlsanity.com/paper/2507.09990/) (2025)
- [Differentially Private Partial Set Cover with Applications to Facility Location](https://mlsanity.com/paper/2207.10240/) (2022)
- [SA-DPSGD: Differentially Private Stochastic Gradient Descent based on Simulated Annealing](https://mlsanity.com/paper/2211.07218/) (2022)
- [DP-SGD with weight clipping](https://mlsanity.com/paper/2310.18001/) (2023)
- [Correlating Cross-Iteration Noise for DP-SGD using Model Curvature](https://mlsanity.com/paper/2510.05416/) (2025)
- [Differentially Private Auditing Under Strategic Response](https://mlsanity.com/paper/2605.07674/) (2026)
- [Encoded Gradients Aggregation against Gradient Leakage in Federated Learning](https://mlsanity.com/paper/2205.13216/) (2022)
- [CURATE: Scaling-up Differentially Private Causal Graph Discovery](https://mlsanity.com/paper/2409.19060/) (2024)
- [Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)](https://mlsanity.com/paper/2508.15141/) (2025)
- [Rethinking Layer-wise Gaussian Noise Injection: Bridging Implicit Objectives and Privacy Budget Allocation](https://mlsanity.com/paper/2509.04232/) (2025)
- [An exponential mechanism based on quadratic approximations for fine-tuning machine learning models with privacy guarantees](https://mlsanity.com/paper/2605.20521/) (2026)

## Abstract (arXiv)

Iterative algorithms, like gradient descent, are common tools for solving a variety of problems, such as model fitting. For this reason, there is interest in creating differentially private versions of them. However, their conversion to differentially private algorithms is often naive. For instance, a fixed number of iterations are chosen, the privacy budget is split evenly among them, and at each iteration, parameters are updated with a noisy gradient. In this paper, we show that gradient-based algorithms can be improved by a more careful allocation of privacy budget per iteration. Intuitively, at the beginning of the optimization, gradients are expected to be large, so that they do not need to be measured as accurately. However, as the parameters approach their optimal values, the gradients decrease and hence need to be measured more accurately. We add a basic line-search capability that helps the algorithm decide when more accurate gradient measurements are necessary. Our gradient descent algorithm works with the recently introduced zCDP version of differential privacy. It outperforms prior algorithms for model fitting and is competitive with the state-of-the-art for (ε,δ)-differential privacy, a strictly weaker definition than zCDP.