# On the Convergence of DP-SGD with Adaptive Clipping

arXiv: https://arxiv.org/abs/2412.19916 · HTML: https://mlsanity.com/paper/2412.19916/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 27 references that are in the map.
- First posted on arXiv 27 Dec 2024.

## Builds on (most cited first, 2 of 2)

- [Toward Training at ImageNet Scale with Differential Privacy](https://mlsanity.com/paper/2201.12328/) (2022)
- [Federated Learning of Gboard Language Models with Differential Privacy](https://mlsanity.com/paper/2305.18465/) (2023)

## Built on by (most cited first, 6 of 6)

- [Smoothed Normalization for Efficient Distributed Private Optimization](https://mlsanity.com/paper/2502.13482/) (2025)
- [Mitigating Disparate Impact of Differentially Private Learning through Bounded Adaptive Clipping](https://mlsanity.com/paper/2506.01396/) (2025)
- [Gradient Shaping Beyond Clipping: A Functional Perspective on Update Magnitude Control](https://mlsanity.com/paper/2510.01578/) (2025)
- [First Provable Guarantees for Practical Private FL: Beyond Restrictive Assumptions](https://mlsanity.com/paper/2512.21521/) (2025)
- [Fast Fourier Transform-Based Spectral and Temporal Gradient Filtering for Differential Privacy](https://mlsanity.com/paper/2505.04468/) (2025)
- [Differentially Private Relational Learning with Entity-level Privacy Guarantees](https://mlsanity.com/paper/2506.08347/) (2025)

## Abstract (arXiv)

Stochastic Gradient Descent (SGD) with gradient clipping is a powerful technique for enabling differentially private optimization. Although prior works extensively investigated clipping with a constant threshold, private training remains highly sensitive to threshold selection, which can be expensive or even infeasible to tune. This sensitivity motivates the development of adaptive approaches, such as quantile clipping, which have demonstrated empirical success but lack a solid theoretical understanding. This paper provides the first comprehensive convergence analysis of SGD with quantile clipping (QC-SGD). We demonstrate that QC-SGD suffers from a bias problem similar to constant-threshold clipped SGD but show how this can be mitigated through a carefully designed quantile and step size schedule. Our analysis reveals crucial relationships between quantile selection, step size, and convergence behavior, providing practical guidelines for parameter selection. We extend these results to differentially private optimization, establishing the first theoretical guarantees for DP-QC-SGD. Our findings provide theoretical foundations for widely used adaptive clipping heuristic and highlight open avenues for future research.