# Batch Clipping and Adaptive Layerwise Clipping for Differential Private Stochastic Gradient Descent

arXiv: https://arxiv.org/abs/2307.11939 · HTML: https://mlsanity.com/paper/2307.11939/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 20 references that are in the map.
- First posted on arXiv 21 Jul 2023.

## Builds on (most cited first, 8 of 8)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Renyi Differential Privacy](https://mlsanity.com/paper/1702.07476/) (2017)
- [Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds](https://mlsanity.com/paper/1605.02065/) (2016)
- [Concentrated Differential Privacy](https://mlsanity.com/paper/1603.01887/) (2016)
- [Three Tools for Practical Differential Privacy](https://mlsanity.com/paper/1812.02890/) (2018)
- [Differentially Private Releasing via Deep Generative Model (Technical Report)](https://mlsanity.com/paper/1801.01594/) (2018)

## Built on by (most cited first, 3 of 3)

- [DP-TLDM: Differentially Private Tabular Latent Diffusion Model](https://mlsanity.com/paper/2403.07842/) (2024)
- [Differentially Private Block-wise Gradient Shuffle for Deep Learning](https://mlsanity.com/paper/2407.21347/) (2024)
- [Sequential Subspace Noise Injection Prevents Accuracy Collapse in Certified Unlearning](https://mlsanity.com/paper/2601.05134/) (2026)

## Abstract (arXiv)

Each round in Differential Private Stochastic Gradient Descent (DPSGD) transmits a sum of clipped gradients obfuscated with Gaussian noise to a central server which uses this to update a global model which often represents a deep neural network. Since the clipped gradients are computed separately, which we call Individual Clipping (IC), deep neural networks like resnet-18 cannot use Batch Normalization Layers (BNL) which is a crucial component in deep neural networks for achieving a high accuracy. To utilize BNL, we introduce Batch Clipping (BC) where, instead of clipping single gradients as in the orginal DPSGD, we average and clip batches of gradients. Moreover, the model entries of different layers have different sensitivities to the added Gaussian noise. Therefore, Adaptive Layerwise Clipping methods (ALC), where each layer has its own adaptively finetuned clipping constant, have been introduced and studied, but so far without rigorous DP proofs. In this paper, we propose em a new ALC and provide rigorous DP proofs for both BC and ALC. Experiments show that our modified DPSGD with BC and ALC for CIFAR-10 with resnet-18 converges while DPSGD with IC and ALC does not.