Calibration Error Estimation Using Fuzzy Binning
Not on the map. Why?
Abstract (arXiv)
Neural network-based decisions tend to be overconfident, where their raw outcome probabilities do not align with the true decision probabilities. Calibration of neural networks is an essential step towards more reliable deep learning frameworks. Prior metrics of calibration error primarily utilize crisp bin membership-based measures. This exacerbates skew in model probabilities and portrays an incomplete picture of calibration error. In this work, we propose a Fuzzy Calibration Error metric (FCE) that utilizes a fuzzy binning approach to calculate calibration error. This approach alleviates the impact of probability skew and provides a tighter estimate while measuring calibration error. We compare our metric with ECE across different data populations and class memberships. Our results show that FCE offers better calibration error estimation, especially in multi-class settings, alleviating the effects of skew in model confidence scores on calibration error estimation. We make our code and supplementary materials available at: https://github.com/bihani-g/fce
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 17 of its 26 references that are in the map. First posted on arXiv 30 Apr 2023.
Builds on
17 of 26 references.
References · 17
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2017On Calibration of Modern Neural Networkscited by 4,860
- 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
- 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
- 2018Predictive Uncertainty Estimation via Prior Networkscited by 555
- 2019Measuring Calibration in Deep Learningcited by 443
- 2020CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instancescited by 441
- 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
- 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 304
- 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
- 2019BERT for Joint Intent Classification and Slot Fillingcited by 195
5 more
- 2020Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learningcited by 190
- 2020Mitigating Bias in Calibration Error Estimationcited by 100
- 2021Low Anisotropy Sense Retrofitting (LASeR) : Towards Isotropic and Sense Enriched Representationscited by 13
- 2023Bag of Tricks for In-Distribution Calibration of Pretrained Transformerscited by 7
- 2021Fuzzy Classification of Multi-intent Utterancescited by 2
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.