# Differentially Private Learning with Per-Sample Adaptive Clipping

arXiv: https://arxiv.org/abs/2212.00328 · HTML: https://mlsanity.com/paper/2212.00328/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 49 references that are in the map.
- First posted on arXiv 1 Dec 2022.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming](https://mlsanity.com/paper/1309.5549/) (2013)
- [Renyi Differential Privacy](https://mlsanity.com/paper/1702.07476/) (2017)
- [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021)
- [Differentially Private Fine-tuning of Language Models](https://mlsanity.com/paper/2110.06500/) (2021)
- [Opacus: User-Friendly Differential Privacy Library in PyTorch](https://mlsanity.com/paper/2109.12298/) (2021)
- [Why gradient clipping accelerates training: A theoretical justification for adaptivity](https://mlsanity.com/paper/1905.11881/) (2019)
- [Differentially Private Learning Needs Better Features (or Much More Data)](https://mlsanity.com/paper/2011.11660/) (2020)
- [Differentially Private Learning with Adaptive Clipping](https://mlsanity.com/paper/1905.03871/) (2019)
- [Differentially Private Empirical Risk Minimization Revisited: Faster and More General](https://mlsanity.com/paper/1802.05251/) (2018)
- [Tempered Sigmoid Activations for Deep Learning with Differential Privacy](https://mlsanity.com/paper/2007.14191/) (2020)
- [Hyperparameter Tuning with Renyi Differential Privacy](https://mlsanity.com/paper/2110.03620/) (2021)
- [Improved Analysis of Clipping Algorithms for Non-convex Optimization](https://mlsanity.com/paper/2010.02519/) (2020)
- [Large Scale Private Learning via Low-rank Reparametrization](https://mlsanity.com/paper/2106.09352/) (2021)
- [Automatic Clipping: Differentially Private Deep Learning Made Easier and Stronger](https://mlsanity.com/paper/2206.07136/) (2022)
- [Toward Training at ImageNet Scale with Differential Privacy](https://mlsanity.com/paper/2201.12328/) (2022)
- [AdaCliP: Adaptive Clipping for Private SGD](https://mlsanity.com/paper/1908.07643/) (2019)
- [Private Adaptive Gradient Methods for Convex Optimization](https://mlsanity.com/paper/2106.13756/) (2021)
- [Differentially Private Stochastic Optimization: New Results in Convex and Non-Convex Settings](https://mlsanity.com/paper/2107.05585/) (2021)
- [Private Adaptive Optimization with Side Information](https://mlsanity.com/paper/2202.05963/) (2022)
- [Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimization](https://mlsanity.com/paper/2206.13033/) (2022)
- [Disparate Impact in Differential Privacy from Gradient Misalignment](https://mlsanity.com/paper/2206.07737/) (2022)

## Built on by (most cited first, 21 of 21)

- [Variance-reduced Clipping for Non-convex Optimization](https://mlsanity.com/paper/2303.00883/) (2023)
- [Differentially Private Natural Language Models: Recent Advances and Future Directions](https://mlsanity.com/paper/2301.09112/) (2023)
- [Augment then Smooth: Reconciling Differential Privacy with Certified Robustness](https://mlsanity.com/paper/2306.08656/) (2023)
- [Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tails](https://mlsanity.com/paper/2405.17529/) (2024)
- [Parameter-Efficient Fine-Tuning with Differential Privacy for Robust Instruction Adaptation in Large Language Models](https://mlsanity.com/paper/2512.06711/) (2025)
- [Defending Against Data Reconstruction Attacks in Federated Learning: An Information Theory Approach](https://mlsanity.com/paper/2403.01268/) (2024)
- [PCDP-SGD: Improving the Convergence of Differentially Private SGD via Projection in Advance](https://mlsanity.com/paper/2312.03792/) (2023)
- [ALI-DPFL: Differentially Private Federated Learning with Adaptive Local Iterations](https://mlsanity.com/paper/2308.10457/) (2023)
- [DP-DCAN: Differentially Private Deep Contrastive Autoencoder Network for Single-cell Clustering](https://mlsanity.com/paper/2311.03410/) (2023)
- [Enhancing DP-SGD through Non-monotonous Adaptive Scaling Gradient Weight](https://mlsanity.com/paper/2411.03059/) (2024)
- [Efficient Differentially Private Fine-Tuning of LLMs via Reinforcement Learning](https://mlsanity.com/paper/2507.22565/) (2025)
- [SoftAdaClip: A Smooth Clipping Strategy for Fair and Private Model Training](https://mlsanity.com/paper/2510.01447/) (2025)
- [When Gradient Clipping Becomes a Control Mechanism for Differential Privacy in Deep Learning](https://mlsanity.com/paper/2602.10584/) (2026)
- [Robust and Fast Training via Per-Sample Clipping](https://mlsanity.com/paper/2605.02701/) (2026)
- [FIBER: A Differentially Private Optimizer with Filter-Aware Innovation Bias Correction](https://mlsanity.com/paper/2605.03425/) (2026)
- [DP-SGD-Global-Adapt-V2-S: Triad Improvements of Privacy, Accuracy and Fairness via Step Decay Noise Multiplier and Step Decay Upper Clipping Threshold](https://mlsanity.com/paper/2312.02400/) (2023)
- [Nosy Layers, Noisy Fixes: Tackling DRAs in Federated Learning Systems using Explainable AI](https://mlsanity.com/paper/2505.10942/) (2025)
- [Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)](https://mlsanity.com/paper/2508.15141/) (2025)
- [Mitigating Membership Inference in Intermediate Representations with Differentially Private Training](https://mlsanity.com/paper/2602.22611/) (2026)
- [TIGER: Inverting Transformer Gradients via Embedding-Subspace Distance Optimization](https://mlsanity.com/paper/2606.18312/) (2026)
- [From Gradient Clipping to Structural Refinement: Improving DPSGD for Medical Image Segmentation](https://mlsanity.com/paper/2606.21763/) (2026)

## Abstract (arXiv)

Privacy in AI remains a topic that draws attention from researchers and the general public in recent years. As one way to implement privacy-preserving AI, differentially private learning is a framework that enables AI models to use differential privacy (DP). To achieve DP in the learning process, existing algorithms typically limit the magnitude of gradients with a constant clipping, which requires carefully tuned due to its significant impact on model performance. As a solution to this issue, latest works NSGD and Auto-S innovatively propose to use normalization instead of clipping to avoid hyperparameter tuning. However, normalization-based approaches like NSGD and Auto-S rely on a monotonic weight function, which imposes excessive weight on small gradient samples and introduces extra deviation to the update. In this paper, we propose a Differentially Private Per-Sample Adaptive Clipping (DP-PSAC) algorithm based on a non-monotonic adaptive weight function, which guarantees privacy without the typical hyperparameter tuning process of using a constant clipping while significantly reducing the deviation between the update and true batch-averaged gradient. We provide a rigorous theoretical convergence analysis and show that with convergence rate at the same order, the proposed algorithm achieves a lower non-vanishing bound, which is maintained over training iterations, compared with NSGD/Auto-S. In addition, through extensive experimental evaluation, we show that DP-PSAC outperforms or matches the state-of-the-art methods on multiple main-stream vision and language tasks.