# Calibrating Deep Neural Networks using Explicit Regularisation and Dynamic Data Pruning

arXiv: https://arxiv.org/abs/2212.10005 · HTML: https://mlsanity.com/paper/2212.10005/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 20 Dec 2022.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibration](https://mlsanity.com/paper/2203.13834/) (2022)
- [Improved Trainable Calibration Method for Neural Networks on Medical Imaging Classification](https://mlsanity.com/paper/2009.04057/) (2020)
- [Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Error](https://mlsanity.com/paper/2106.09613/) (2021)
- [On Deep Neural Network Calibration by Regularization and its Impact on Refinement](https://mlsanity.com/paper/2106.09385/) (2021)

## Built on by (most cited first, 18 of 18)

- [Calibration in Deep Learning: A Survey of the State-of-the-Art](https://mlsanity.com/paper/2308.01222/) (2023)
- [ACLS: Adaptive and Conditional Label Smoothing for Network Calibration](https://mlsanity.com/paper/2308.11911/) (2023)
- [IKD+: Reliable Low Complexity Deep Models For Retinopathy Classification](https://mlsanity.com/paper/2303.02310/) (2023)
- [Unlocking the Potential of Model Calibration in Federated Learning](https://mlsanity.com/paper/2409.04901/) (2024)
- [Rethinking Semi-supervised Segmentation Beyond Accuracy: Reliability and Robustness](https://mlsanity.com/paper/2506.05917/) (2025)
- [h-calibration: Rethinking Classifier Recalibration with Probabilistic Error-Bounded Objective](https://mlsanity.com/paper/2506.17968/) (2025)
- [Prompting without Panic: Attribute-aware, Zero-shot, Test-Time Calibration](https://mlsanity.com/paper/2506.22819/) (2025)
- [Confidence-Based Response Abstinence: Improving LLM Trustworthiness via Activation-Based Uncertainty Estimation](https://mlsanity.com/paper/2510.13750/) (2025)
- [Towards Unbiased Calibration using Meta-Regularization](https://mlsanity.com/paper/2303.15057/) (2023)
- [Conservative Predictions on Noisy Financial Data](https://mlsanity.com/paper/2310.11815/) (2023)
- [Reducing the False Positive Rate Using Bayesian Inference in Autonomous Driving Perception](https://mlsanity.com/paper/2310.05951/) (2023)
- [Pseudo-label Learning with Calibrated Confidence Using an Energy-based Model](https://mlsanity.com/paper/2404.09585/) (2024)
- [Accurate and Reliable Predictions with Mutual-Transport Ensemble](https://mlsanity.com/paper/2405.19656/) (2024)
- [Dual-View Pyramid Pooling in Deep Neural Networks for Improved Medical Image Classification and Confidence Calibration](https://mlsanity.com/paper/2408.02906/) (2024)
- [Adaptive Confidence-Wise Loss for Improved Lens Structure Segmentation in AS-OCT](https://mlsanity.com/paper/2508.08705/) (2025)
- [SCOPE: Semantic Coreset with Orthogonal Projection Embeddings for Federated learning](https://mlsanity.com/paper/2603.12976/) (2026)
- [Enhancing Deep Neural Network Reliability with Refinement and Calibration](https://mlsanity.com/paper/2605.23249/) (2026)
- [The Impact of CutMix on Reliability and Robustness in Semantic Segmentation](https://mlsanity.com/paper/2608.18715/) (2026)

## Abstract (arXiv)

Deep neural networks (DNN) are prone to miscalibrated predictions, often exhibiting a mismatch between the predicted output and the associated confidence scores. Contemporary model calibration techniques mitigate the problem of overconfident predictions by pushing down the confidence of the winning class while increasing the confidence of the remaining classes across all test samples. However, from a deployment perspective, an ideal model is desired to (i) generate well-calibrated predictions for high-confidence samples with predicted probability say >0.95, and (ii) generate a higher proportion of legitimate high-confidence samples. To this end, we propose a novel regularization technique that can be used with classification losses, leading to state-of-the-art calibrated predictions at test time; From a deployment standpoint in safety-critical applications, only high-confidence samples from a well-calibrated model are of interest, as the remaining samples have to undergo manual inspection. Predictive confidence reduction of these potentially ``high-confidence samples'' is a downside of existing calibration approaches. We mitigate this by proposing a dynamic train-time data pruning strategy that prunes low-confidence samples every few epochs, providing an increase in "confident yet calibrated samples". We demonstrate state-of-the-art calibration performance across image classification benchmarks, reducing training time without much compromise in accuracy. We provide insights into why our dynamic pruning strategy that prunes low-confidence training samples leads to an increase in high-confidence samples at test time.