# Training Private Models That Know What They Don't Know

arXiv: https://arxiv.org/abs/2305.18393 · HTML: https://mlsanity.com/paper/2305.18393/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 28 May 2023.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [Renyi Differential Privacy](https://mlsanity.com/paper/1702.07476/) (2017)
- [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://mlsanity.com/paper/2006.10108/) (2020)
- [Opacus: User-Friendly Differential Privacy Library in PyTorch](https://mlsanity.com/paper/2109.12298/) (2021)
- [SelectiveNet: A Deep Neural Network with an Integrated Reject Option](https://mlsanity.com/paper/1901.09192/) (2019)
- [Differential Privacy Has Disparate Impact on Model Accuracy](https://mlsanity.com/paper/1905.12101/) (2019)
- [Consistent Estimators for Learning to Defer to an Expert](https://mlsanity.com/paper/2006.01862/) (2020)
- [Self-Adaptive Training: beyond Empirical Risk Minimization](https://mlsanity.com/paper/2002.10319/) (2020)
- [Finite Sample Differentially Private Confidence Intervals](https://mlsanity.com/paper/1711.03908/) (2017)
- [Deep Gamblers: Learning to Abstain with Portfolio Theory](https://mlsanity.com/paper/1907.00208/) (2019)
- [Selective Classification Can Magnify Disparities Across Groups](https://mlsanity.com/paper/2010.14134/) (2020)
- [Regression with reject option and application to kNN](https://mlsanity.com/paper/2006.16597/) (2020)
- [Towards Better Selective Classification](https://mlsanity.com/paper/2206.09034/) (2022)
- [Generating and Characterizing Scenarios for Safety Testing of Autonomous Vehicles](https://mlsanity.com/paper/2103.07403/) (2021)
- [Selective Prediction via Training Dynamics](https://mlsanity.com/paper/2205.13532/) (2022)
- [Parametric Bootstrap for Differentially Private Confidence Intervals](https://mlsanity.com/paper/2006.07749/) (2020)
- [Unbiased Statistical Estimation and Valid Confidence Intervals Under Differential Privacy](https://mlsanity.com/paper/2110.14465/) (2021)

## Built on by (most cited first, 5 of 5)

- [Private Gradient Descent for Linear Regression: Tighter Error Bounds and Instance-Specific Uncertainty Estimation](https://mlsanity.com/paper/2402.13531/) (2024)
- [Gradients Look Alike: Sensitivity is Often Overestimated in DP-SGD](https://mlsanity.com/paper/2307.00310/) (2023)
- [Gatekeeper: Improving Model Cascades Through Confidence Tuning](https://mlsanity.com/paper/2502.19335/) (2025)
- [What Does It Take to Build a Performant Selective Classifier?](https://mlsanity.com/paper/2510.20242/) (2025)
- [Better than Average: Spatially-Aware Aggregation of Segmentation Uncertainty Improves Downstream Performance](https://mlsanity.com/paper/2603.29941/) (2026)

## Abstract (arXiv)

Training reliable deep learning models which avoid making overconfident but incorrect predictions is a longstanding challenge. This challenge is further exacerbated when learning has to be differentially private: protection provided to sensitive data comes at the price of injecting additional randomness into the learning process. In this work, we conduct a thorough empirical investigation of selective classifiers -- that can abstain when they are unsure -- under a differential privacy constraint. We find that several popular selective prediction approaches are ineffective in a differentially private setting as they increase the risk of privacy leakage. At the same time, we identify that a recent approach that only uses checkpoints produced by an off-the-shelf private learning algorithm stands out as particularly suitable under DP. Further, we show that differential privacy does not just harm utility but also degrades selective classification performance. To analyze this effect across privacy levels, we propose a novel evaluation mechanism which isolate selective prediction performance across model utility levels. Our experimental results show that recovering the performance level attainable by non-private models is possible but comes at a considerable coverage cost as the privacy budget decreases.