# Differentially private training of neural networks with Langevin dynamics for calibrated predictive uncertainty

arXiv: https://arxiv.org/abs/2107.04296 · HTML: https://mlsanity.com/paper/2107.04296/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 9 Jul 2021.

## Builds on (most cited first, 12 of 12)

- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Understanding Gradient Clipping in Private SGD: A Geometric Perspective](https://mlsanity.com/paper/2006.15429/) (2020)
- [The Case for Bayesian Deep Learning](https://mlsanity.com/paper/2001.10995/) (2020)
- [Privacy for Free: Posterior Sampling and Stochastic Gradient Monte Carlo](https://mlsanity.com/paper/1502.07645/) (2015)
- [On Connecting Stochastic Gradient MCMC and Differential Privacy](https://mlsanity.com/paper/1712.09097/) (2017)
- [Stochastic Gradient Langevin Dynamics Algorithms with Adaptive Drifts](https://mlsanity.com/paper/2009.09535/) (2020)

## Built on by (most cited first, 2 of 2)

- [A Closer Look at the Calibration of Differentially Private Learners](https://mlsanity.com/paper/2210.08248/) (2022)
- [Equivariant Differentially Private Deep Learning: Why DP-SGD Needs Sparser Models](https://mlsanity.com/paper/2301.13104/) (2023)

## Abstract (arXiv)

We show that differentially private stochastic gradient descent (DP-SGD) can yield poorly calibrated, overconfident deep learning models. This represents a serious issue for safety-critical applications, e.g. in medical diagnosis. We highlight and exploit parallels between stochastic gradient Langevin dynamics, a scalable Bayesian inference technique for training deep neural networks, and DP-SGD, in order to train differentially private, Bayesian neural networks with minor adjustments to the original (DP-SGD) algorithm. Our approach provides considerably more reliable uncertainty estimates than DP-SGD, as demonstrated empirically by a reduction in expected calibration error (MNIST ~5-fold, Pediatric Pneumonia Dataset ~2-fold).