# Efficient Per-Example Gradient Computations in Convolutional Neural Networks

arXiv: https://arxiv.org/abs/1912.06015 · HTML: https://mlsanity.com/paper/1912.06015/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Dec 2019.

## Built on by (most cited first, 17 of 17)

- [Opacus: User-Friendly Differential Privacy Library in PyTorch](https://mlsanity.com/paper/2109.12298/) (2021)
- [Data Shapley in One Training Run](https://mlsanity.com/paper/2406.11011/) (2024)
- [Differentially Private Optimization on Large Model at Small Cost](https://mlsanity.com/paper/2210.00038/) (2022)
- [Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacy](https://mlsanity.com/paper/2205.10683/) (2022)
- [Fast and Memory Efficient Differentially Private-SGD via JL Projections](https://mlsanity.com/paper/2102.03013/) (2021)
- [Model-agnostic out-of-distribution detection using combined statistical tests](https://mlsanity.com/paper/2203.01097/) (2022)
- [Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers](https://mlsanity.com/paper/2411.00999/) (2024)
- [Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliability](https://mlsanity.com/paper/2107.08461/) (2021)
- [Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimization](https://mlsanity.com/paper/2406.08001/) (2024)
- [DP-MemArc: Differential Privacy Transfer Learning for Memory Efficient Language Models](https://mlsanity.com/paper/2406.11087/) (2024)
- [Rieoptax: Riemannian Optimization in JAX](https://mlsanity.com/paper/2210.04840/) (2022)
- [Noise is All You Need: Private Second-Order Convergence of Noisy SGD](https://mlsanity.com/paper/2410.06878/) (2024)
- [FlashDP: Private Training Large Language Models with Efficient DP-SGD](https://mlsanity.com/paper/2507.01154/) (2025)
- [Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tail](https://mlsanity.com/paper/2605.31244/) (2026)
- [Differentially Private Optimization for Non-Decomposable Objective Functions](https://mlsanity.com/paper/2310.03104/) (2023)
- [Reducing Per-Sample Interference in Stochastic Optimization](https://mlsanity.com/paper/2607.16261/) (2026)
- [Is Weight Tying Still Beneficial for Decoder-Only LLMs in Private Settings Under DP-SGD?](https://mlsanity.com/paper/2609.40335/) (2026)

## Abstract (arXiv)

Deep learning frameworks leverage GPUs to perform massively-parallel computations over batches of many training examples efficiently. However, for certain tasks, one may be interested in performing per-example computations, for instance using per-example gradients to evaluate a quantity of interest unique to each example. One notable application comes from the field of differential privacy, where per-example gradients must be norm-bounded in order to limit the impact of each example on the aggregated batch gradient. In this work, we discuss how per-example gradients can be efficiently computed in convolutional neural networks (CNNs). We compare existing strategies by performing a few steps of differentially-private training on CNNs of varying sizes. We also introduce a new strategy for per-example gradient calculation, which is shown to be advantageous depending on the model architecture and how the model is trained. This is a first step in making differentially-private training of CNNs practical.