# Efficient Per-Example Gradient Computations

arXiv: https://arxiv.org/abs/1510.01799 · HTML: https://mlsanity.com/paper/1510.01799/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2015.

## Built on by (most cited first, 30 of 42)

- [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021)
- [Opacus: User-Friendly Differential Privacy Library in PyTorch](https://mlsanity.com/paper/2109.12298/) (2021)
- [How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy](https://mlsanity.com/paper/2303.00654/) (2023)
- [Deep Learning with Label Differential Privacy](https://mlsanity.com/paper/2102.06062/) (2021)
- [Toward Training at ImageNet Scale with Differential Privacy](https://mlsanity.com/paper/2201.12328/) (2022)
- [Differentially Private Optimization on Large Model at Small Cost](https://mlsanity.com/paper/2210.00038/) (2022)
- [Exploring the Limits of Differentially Private Deep Learning with Group-wise Clipping](https://mlsanity.com/paper/2212.01539/) (2022)
- [Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacy](https://mlsanity.com/paper/2205.10683/) (2022)
- [Differentially Private Bias-Term Fine-tuning of Foundation Models](https://mlsanity.com/paper/2210.00036/) (2022)
- [RelaxLoss: Defending Membership Inference Attacks without Losing Utility](https://mlsanity.com/paper/2207.05801/) (2022)
- [Fast and Memory Efficient Differentially Private-SGD via JL Projections](https://mlsanity.com/paper/2102.03013/) (2021)
- [Introspective Learning : A Two-Stage Approach for Inference in Neural Networks](https://mlsanity.com/paper/2209.08425/) (2022)
- [Model-agnostic out-of-distribution detection using combined statistical tests](https://mlsanity.com/paper/2203.01097/) (2022)
- [Differentially Private Image Classification from Features](https://mlsanity.com/paper/2211.13403/) (2022)
- [Zero redundancy distributed learning with differential privacy](https://mlsanity.com/paper/2311.11822/) (2023)
- [Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers](https://mlsanity.com/paper/2411.00999/) (2024)
- [Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliability](https://mlsanity.com/paper/2107.08461/) (2021)
- [Private Ad Modeling with DP-SGD](https://mlsanity.com/paper/2211.11896/) (2022)
- [Private Learning with Public Features](https://mlsanity.com/paper/2310.15454/) (2023)
- [Delving into Differentially Private Transformer](https://mlsanity.com/paper/2405.18194/) (2024)
- [Learning Deep Neural Networks under Agnostic Corrupted Supervision](https://mlsanity.com/paper/2102.06735/) (2021)
- [Bayesian Uncertainty and Expected Gradient Length -- Regression: Two Sides Of The Same Coin?](https://mlsanity.com/paper/2104.09493/) (2021)
- [R&B: Domain Regrouping and Data Mixture Balancing for Efficient Foundation Model Training](https://mlsanity.com/paper/2505.00358/) (2025)
- [Transformer with Memory Replay](https://mlsanity.com/paper/2205.09869/) (2022)
- [DPFormer: Learning Differentially Private Transformer on Long-Tailed Data](https://mlsanity.com/paper/2305.17633/) (2023)
- [Differentially Private Training of Mixture of Experts Models](https://mlsanity.com/paper/2402.07334/) (2024)
- [DP-MemArc: Differential Privacy Transfer Learning for Memory Efficient Language Models](https://mlsanity.com/paper/2406.11087/) (2024)
- [Learning with Differentially Private (Sliced) Wasserstein Gradients](https://mlsanity.com/paper/2502.01701/) (2025)
- [Per-example gradients: a new frontier for understanding and improving optimizers](https://mlsanity.com/paper/2510.00236/) (2025)
- [Private Cross-Silo Federated Learning for Extracting Vaccine Adverse Event Mentions](https://mlsanity.com/paper/2103.07491/) (2021)

## Abstract (arXiv)

This technical report describes an efficient technique for computing the norm of the gradient of the loss function for a neural network with respect to its parameters. This gradient norm can be computed efficiently for every example.