# A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibration

arXiv: https://arxiv.org/abs/2203.13834 · HTML: https://mlsanity.com/paper/2203.13834/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 59 references that are in the map.
- First posted on arXiv 25 Mar 2022.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem](https://mlsanity.com/paper/1812.05720/) (2018)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks](https://mlsanity.com/paper/1905.11001/) (2019)
- [Learning Confidence for Out-of-Distribution Detection in Neural Networks](https://mlsanity.com/paper/1802.04865/) (2018)
- [Local Temperature Scaling for Probability Calibration](https://mlsanity.com/paper/2008.05105/) (2020)
- [Calibration tests in multi-class classification: A unifying framework](https://mlsanity.com/paper/1910.11385/) (2019)
- [Post-hoc Uncertainty Calibration for Domain Drift Scenarios](https://mlsanity.com/paper/2012.10988/) (2020)

## Built on by (most cited first, 30 of 49)

- [Calibration in Deep Learning: A Survey of the State-of-the-Art](https://mlsanity.com/paper/2308.01222/) (2023)
- [Rethinking Confidence Calibration for Failure Prediction](https://mlsanity.com/paper/2303.02970/) (2023)
- [Towards Improving Calibration in Object Detection Under Domain Shift](https://mlsanity.com/paper/2209.07601/) (2022)
- [Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator](https://mlsanity.com/paper/2505.16690/) (2025)
- [RankMixup: Ranking-Based Mixup Training for Network Calibration](https://mlsanity.com/paper/2308.11990/) (2023)
- [Calibrating Deep Neural Networks using Explicit Regularisation and Dynamic Data Pruning](https://mlsanity.com/paper/2212.10005/) (2022)
- [Multiclass Confidence and Localization Calibration for Object Detection](https://mlsanity.com/paper/2306.08271/) (2023)
- [ACLS: Adaptive and Conditional Label Smoothing for Network Calibration](https://mlsanity.com/paper/2308.11911/) (2023)
- [O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Models](https://mlsanity.com/paper/2503.12096/) (2025)
- [Dynamic Correlation Learning and Regularization for Multi-Label Confidence Calibration](https://mlsanity.com/paper/2407.06844/) (2024)
- [Are foundation models for computer vision good conformal predictors?](https://mlsanity.com/paper/2412.06082/) (2024)
- [Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection](https://mlsanity.com/paper/2303.14404/) (2023)
- [Transfer Knowledge from Head to Tail: Uncertainty Calibration under Long-tailed Distribution](https://mlsanity.com/paper/2304.06537/) (2023)
- [Cal-DETR: Calibrated Detection Transformer](https://mlsanity.com/paper/2311.03570/) (2023)
- [On Calibration of Object Detectors: Pitfalls, Evaluation and Baselines](https://mlsanity.com/paper/2405.20459/) (2024)
- [Revisiting Confidence Estimation: Towards Reliable Failure Prediction](https://mlsanity.com/paper/2403.02886/) (2024)
- [Beyond Classification: Definition and Density-based Estimation of Calibration in Object Detection](https://mlsanity.com/paper/2312.06645/) (2023)
- [Multi-Head Multi-Loss Model Calibration](https://mlsanity.com/paper/2303.01099/) (2023)
- [Threshold-Consistent Margin Loss for Open-World Deep Metric Learning](https://mlsanity.com/paper/2307.04047/) (2023)
- [A Novel Data Augmentation Technique for Out-of-Distribution Sample Detection using Compounded Corruptions](https://mlsanity.com/paper/2207.13916/) (2022)
- [Average Calibration Error: A Differentiable Loss for Improved Reliability in Image Segmentation](https://mlsanity.com/paper/2403.06759/) (2024)
- [Performance Control in Early Exiting to Deploy Large Models at the Same Cost of Smaller Ones](https://mlsanity.com/paper/2412.19325/) (2024)
- [AUGCAL: Improving Sim2Real Adaptation by Uncertainty Calibration on Augmented Synthetic Images](https://mlsanity.com/paper/2312.06106/) (2023)
- [The Misclassification Likelihood Matrix: Some Classes Are More Likely To Be Misclassified Than Others](https://mlsanity.com/paper/2407.07818/) (2024)
- [Unlocking the Potential of Model Calibration in Federated Learning](https://mlsanity.com/paper/2409.04901/) (2024)
- [Ordinal-Aware Calibration for Ordinal Classification](https://mlsanity.com/paper/2410.15658/) (2024)
- [Average Calibration Losses for Reliable Uncertainty in Medical Image Segmentation](https://mlsanity.com/paper/2506.03942/) (2025)
- [Sample Margin-Aware Recalibration of Temperature Scaling](https://mlsanity.com/paper/2506.23492/) (2025)
- [Calibration-Aware Prompt Learning for Medical Vision-Language Models](https://mlsanity.com/paper/2509.15226/) (2025)
- [Consistency-Guided Temperature Scaling Using Style and Content Information for Out-of-Domain Calibration](https://mlsanity.com/paper/2402.15019/) (2024)

## Abstract (arXiv)

Deep Neural Networks ( DNN s) are known to make overconfident mistakes, which makes their use problematic in safety-critical applications. State-of-the-art ( SOTA ) calibration techniques improve on the confidence of predicted labels alone and leave the confidence of non-max classes (e.g. top-2, top-5) uncalibrated. Such calibration is not suitable for label refinement using post-processing. Further, most SOTA techniques learn a few hyper-parameters post-hoc, leaving out the scope for image, or pixel specific calibration. This makes them unsuitable for calibration under domain shift, or for dense prediction tasks like semantic segmentation. In this paper, we argue for intervening at the train time itself, so as to directly produce calibrated DNN models. We propose a novel auxiliary loss function: Multi-class Difference in Confidence and Accuracy ( MDCA ), to achieve the same MDCA can be used in conjunction with other application/task-specific loss functions. We show that training with MDCA leads to better-calibrated models in terms of Expected Calibration Error ( ECE ), and Static Calibration Error ( SCE ) on image classification, and segmentation tasks. We report ECE ( SCE ) score of 0.72 (1.60) on the CIFAR 100 dataset, in comparison to 1.90 (1.71) by the SOTA. Under domain shift, a ResNet-18 model trained on PACS dataset using MDCA gives an average ECE ( SCE ) score of 19.7 (9.7) across all domains, compared to 24.2 (11.8) by the SOTA. For the segmentation task, we report a 2X reduction in calibration error on PASCAL - VOC dataset in comparison to Focal Loss. Finally, MDCA training improves calibration even on imbalanced data, and for natural language classification tasks. We have released the code here: code is available at https://github.com/mdca-loss