# Calibrating Deep Neural Networks using Focal Loss

arXiv: https://arxiv.org/abs/2002.09437 · HTML: https://mlsanity.com/paper/2002.09437/

## Key facts

- Cited by 357 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Feb 2020.

## Built on by (most cited first, 30 of 357)

- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [Mitigating Neural Network Overconfidence with Logit Normalization](https://mlsanity.com/paper/2205.09310/) (2022)
- [Improving Calibration for Long-Tailed Recognition](https://mlsanity.com/paper/2104.00466/) (2021)
- [On Calibration and Out-of-domain Generalization](https://mlsanity.com/paper/2102.10395/) (2021)
- [Taming Overconfidence in LLMs: Reward Calibration in RLHF](https://mlsanity.com/paper/2410.09724/) (2024)
- [Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis](https://mlsanity.com/paper/2210.04714/) (2022)
- [Evidential Deep Learning for Open Set Action Recognition](https://mlsanity.com/paper/2107.10161/) (2021)
- [Soft Calibration Objectives for Neural Networks](https://mlsanity.com/paper/2108.00106/) (2021)
- [The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration](https://mlsanity.com/paper/2111.15430/) (2021)
- [Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly](https://mlsanity.com/paper/2204.13631/) (2022)
- [C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion](https://mlsanity.com/paper/2403.14119/) (2024)
- [Are All Losses Created Equal: A Neural Collapse Perspective](https://mlsanity.com/paper/2210.02192/) (2022)
- [Calibration in Deep Learning: A Survey of the State-of-the-Art](https://mlsanity.com/paper/2308.01222/) (2023)
- [Training Uncertainty-Aware Classifiers with Conformalized Deep Learning](https://mlsanity.com/paper/2205.05878/) (2022)
- [Test-Time Adaptation via Self-Training with Nearest Neighbor Information](https://mlsanity.com/paper/2207.10792/) (2022)
- [Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts](https://mlsanity.com/paper/2305.19951/) (2023)
- [A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibration](https://mlsanity.com/paper/2203.13834/) (2022)
- [Sample-dependent Adaptive Temperature Scaling for Improved Calibration](https://mlsanity.com/paper/2207.06211/) (2022)
- [A Consistent and Differentiable Lp Canonical Calibration Error Estimator](https://mlsanity.com/paper/2210.07810/) (2022)
- [Top-label calibration and multiclass-to-binary reductions](https://mlsanity.com/paper/2107.08353/) (2021)
- [BERT & Family Eat Word Salad: Experiments with Text Understanding](https://mlsanity.com/paper/2101.03453/) (2021)
- [Uncertainty in Natural Language Processing: Sources, Quantification, and Applications](https://mlsanity.com/paper/2306.04459/) (2023)
- [Neural Collapse Inspired Attraction-Repulsion-Balanced Loss for Imbalanced Learning](https://mlsanity.com/paper/2204.08735/) (2022)
- [Rethinking Confidence Calibration for Failure Prediction](https://mlsanity.com/paper/2303.02970/) (2023)
- [Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification](https://mlsanity.com/paper/2102.07856/) (2021)
- [Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributions](https://mlsanity.com/paper/2111.08456/) (2021)
- [OpenMix: Exploring Outlier Samples for Misclassification Detection](https://mlsanity.com/paper/2303.17093/) (2023)
- [Thermometer: Towards Universal Calibration for Large Language Models](https://mlsanity.com/paper/2403.08819/) (2024)
- [BEARS Make Neuro-Symbolic Models Aware of their Reasoning Shortcuts](https://mlsanity.com/paper/2402.12240/) (2024)

## Abstract (arXiv)

Miscalibration - a mismatch between a model's confidence and its correctness - of Deep Neural Networks (DNNs) makes their predictions hard to rely on. Ideally, we want networks to be accurate, calibrated and confident. We show that, as opposed to the standard cross-entropy loss, focal loss [Lin et. al., 2017] allows us to learn models that are already very well calibrated. When combined with temperature scaling, whilst preserving accuracy, it yields state-of-the-art calibrated models. We provide a thorough analysis of the factors causing miscalibration, and use the insights we glean from this to justify the empirically excellent performance of focal loss. To facilitate the use of focal loss in practice, we also provide a principled approach to automatically select the hyperparameter involved in the loss function. We perform extensive experiments on a variety of computer vision and NLP datasets, and with a wide variety of network architectures, and show that our approach achieves state-of-the-art calibration without compromising on accuracy in almost all cases. Code is available at https://github.com/torrvision/focal_calibration.