# Dynamic Differential-Privacy Preserving SGD

arXiv: https://arxiv.org/abs/2111.00173 · HTML: https://mlsanity.com/paper/2111.00173/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 35 references that are in the map.
- First posted on arXiv 30 Oct 2021.

## Builds on (most cited first, 17 of 17)

- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [Renyi Differential Privacy](https://mlsanity.com/paper/1702.07476/) (2017)
- [iDLG: Improved Deep Leakage from Gradients](https://mlsanity.com/paper/2001.02610/) (2020)
- [Subsampled R'enyi Differential Privacy and Analytical Moments Accountant](https://mlsanity.com/paper/1808.00087/) (2018)
- [Differentially Private Learning with Adaptive Clipping](https://mlsanity.com/paper/1905.03871/) (2019)
- [Numerical Composition of Differential Privacy](https://mlsanity.com/paper/2106.02848/) (2021)
- [Understanding Gradient Clipping in Private SGD: A Geometric Perspective](https://mlsanity.com/paper/2006.15429/) (2020)
- [Deep Learning with Gaussian Differential Privacy](https://mlsanity.com/paper/1911.11607/) (2019)
- [Optimal Accounting of Differential Privacy via Characteristic Function](https://mlsanity.com/paper/2106.08567/) (2021)
- [AdaCliP: Adaptive Clipping for Private SGD](https://mlsanity.com/paper/1908.07643/) (2019)
- [Differentially Private Model Publishing for Deep Learning](https://mlsanity.com/paper/1904.02200/) (2019)
- [Efficient Private ERM for Smooth Objectives](https://mlsanity.com/paper/1703.09947/) (2017)
- [Evading Curse of Dimensionality in Unconstrained Private GLMs via Private Gradient Descent](https://mlsanity.com/paper/2006.06783/) (2020)
- [Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds](https://mlsanity.com/paper/2006.13501/) (2020)
- [Adaptive Differentially Private Empirical Risk Minimization](https://mlsanity.com/paper/2110.07435/) (2021)

## Built on by (most cited first, 30 of 30)

- [Differentially Private Learning with Per-Sample Adaptive Clipping](https://mlsanity.com/paper/2212.00328/) (2022)
- [Beyond Uniform Lipschitz Condition in Differentially Private Optimization](https://mlsanity.com/paper/2206.10713/) (2022)
- [Time-series Transformer Generative Adversarial Networks](https://mlsanity.com/paper/2205.11164/) (2022)
- [DIFF2: Differential Private Optimization via Gradient Differences for Nonconvex Distributed Learning](https://mlsanity.com/paper/2302.03884/) (2023)
- [Convergence and Privacy of Decentralized Nonconvex Optimization with Gradient Clipping and Communication Compression](https://mlsanity.com/paper/2305.09896/) (2023)
- [Differentially Private Video Activity Recognition](https://mlsanity.com/paper/2306.15742/) (2023)
- [Secure Split Learning against Property Inference, Data Reconstruction, and Feature Space Hijacking Attacks](https://mlsanity.com/paper/2304.09515/) (2023)
- [Unlearning with Fisher Masking](https://mlsanity.com/paper/2310.05331/) (2023)
- [DP-FP: Differentially Private Forward Propagation for Large Models](https://mlsanity.com/paper/2112.14430/) (2021)
- [DC-SGD: Differentially Private SGD with Dynamic Clipping through Gradient Norm Distribution Estimation](https://mlsanity.com/paper/2503.22988/) (2025)
- [Secure and Trustworthy Artificial Intelligence-Extended Reality (AI-XR) for Metaverses](https://mlsanity.com/paper/2210.13289/) (2022)
- [Video-DPRP: A Differentially Private Approach for Visual Privacy-Preserving Video Human Activity Recognition](https://mlsanity.com/paper/2503.02132/) (2025)
- [GeoClip: Geometry-Aware Clipping for Differentially Private SGD](https://mlsanity.com/paper/2506.06549/) (2025)
- [Differentially Private Wireless Federated Learning Using Orthogonal Sequences](https://mlsanity.com/paper/2306.08280/) (2023)
- [Enhancing DP-SGD through Non-monotonous Adaptive Scaling Gradient Weight](https://mlsanity.com/paper/2411.03059/) (2024)
- [Federated Learning with Differential Privacy: An Utility-Enhanced Approach](https://mlsanity.com/paper/2503.21154/) (2025)
- [Non-Asymptotic Analysis of Online Local Private Learning with SGD](https://mlsanity.com/paper/2507.07041/) (2025)
- [Q-DPTS: Quantum Differentially Private Time Series Forecasting via Variational Quantum Circuits](https://mlsanity.com/paper/2508.05036/) (2025)
- [Learning to Censor by Noisy Sampling](https://mlsanity.com/paper/2203.12192/) (2022)
- [Multi-Objective Optimization for Privacy-Utility Balance in Differentially Private Federated Learning](https://mlsanity.com/paper/2503.21159/) (2025)
- [Adaptive Clipping for Privacy-Preserving Few-Shot Learning: Enhancing Generalization with Limited Data](https://mlsanity.com/paper/2503.22749/) (2025)
- [Dyn-D2P: Dynamic Differentially Private Decentralized Learning with Provable Utility Guarantee](https://mlsanity.com/paper/2505.06651/) (2025)
- [Heart rate and respiratory rate prediction from noisy real-world smartphone based on Deep Learning methods](https://mlsanity.com/paper/2506.22460/) (2025)
- [Steps Adaptive Decay DPSGD: Enhancing Performance on Imbalanced Datasets with Differential Privacy with HAM10000](https://mlsanity.com/paper/2507.06619/) (2025)
- [Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)](https://mlsanity.com/paper/2508.15141/) (2025)
- [Stabilization of Perturbed Loss Function: Differential Privacy without Gradient Noise](https://mlsanity.com/paper/2508.15523/) (2025)
- [Balancing Utility and Privacy: Dynamically Private SGD with Random Projection](https://mlsanity.com/paper/2509.09485/) (2025)
- [DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models](https://mlsanity.com/paper/2605.10272/) (2026)
- [StraightDP: Geometry-Aware Differential Privacy for Rectified-Flow Transformers](https://mlsanity.com/paper/2607.29100/) (2026)
- [Revisiting the Provable-Auditable Privacy Gap of DP-SGD](https://mlsanity.com/paper/2608.28934/) (2026)

## Abstract (arXiv)

The vanilla Differentially-Private Stochastic Gradient Descent (DP-SGD), including DP-Adam and other variants, ensures the privacy of training data by uniformly distributing privacy costs across training steps. The equivalent privacy costs controlled by maintaining the same gradient clipping thresholds and noise powers in each step result in unstable updates and a lower model accuracy when compared to the non-DP counterpart. In this paper, we propose the dynamic DP-SGD (along with dynamic DP-Adam, and others) to reduce the performance loss gap while maintaining privacy by dynamically adjusting clipping thresholds and noise powers while adhering to a total privacy budget constraint. Extensive experiments on a variety of deep learning tasks, including image classification, natural language processing, and federated learning, demonstrate that the proposed dynamic DP-SGD algorithm stabilizes updates and, as a result, significantly improves model accuracy in the strong privacy protection region when compared to the vanilla DP-SGD. We also conduct theoretical analysis to better understand the privacy-utility trade-off with dynamic DP-SGD, as well as to learn why Dynamic DP-SGD can outperform vanilla DP-SGD.