# AdaFocal: Calibration-aware Adaptive Focal Loss

arXiv: https://arxiv.org/abs/2211.11838 · HTML: https://mlsanity.com/paper/2211.11838/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 41 references that are in the map.
- First posted on arXiv 21 Nov 2022.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning](https://mlsanity.com/paper/2003.07329/) (2020)
- [Evaluating model calibration in classification](https://mlsanity.com/paper/1902.06977/) (2019)
- [Mitigating Bias in Calibration Error Estimation](https://mlsanity.com/paper/2012.08668/) (2020)
- [Calibration of Neural Networks using Splines](https://mlsanity.com/paper/2006.12800/) (2020)
- [Posterior calibration and exploratory analysis for natural language processing models](https://mlsanity.com/paper/1508.05154/) (2015)
- [Binary Classifier Calibration using an Ensemble of Near Isotonic Regression Models](https://mlsanity.com/paper/1511.05191/) (2015)

## Built on by (most cited first, 30 of 32)

- [Dual Focal Loss for Calibration](https://mlsanity.com/paper/2305.13665/) (2023)
- [RankMixup: Ranking-Based Mixup Training for Network Calibration](https://mlsanity.com/paper/2308.11990/) (2023)
- [ACLS: Adaptive and Conditional Label Smoothing for Network Calibration](https://mlsanity.com/paper/2308.11911/) (2023)
- [A Short Survey on Importance Weighting for Machine Learning](https://mlsanity.com/paper/2403.10175/) (2024)
- [Feature Clipping for Uncertainty Calibration](https://mlsanity.com/paper/2410.19796/) (2024)
- [Selective Learning: Towards Robust Calibration with Dynamic Regularization](https://mlsanity.com/paper/2402.08384/) (2024)
- [Variational Supervised Contrastive Learning](https://mlsanity.com/paper/2506.07413/) (2025)
- [Improving Calibration by Relating Focal Loss, Temperature Scaling, and Properness](https://mlsanity.com/paper/2408.11598/) (2024)
- [T-CIL: Temperature Scaling using Adversarial Perturbation for Calibration in Class-Incremental Learning](https://mlsanity.com/paper/2503.22163/) (2025)
- [Model Calibration in Dense Classification with Adaptive Label Perturbation](https://mlsanity.com/paper/2307.13539/) (2023)
- [Balancing Two Classifiers via A Simplex ETF Structure for Model Calibration](https://mlsanity.com/paper/2504.10007/) (2025)
- [Selective classification using a robust meta-learning approach](https://mlsanity.com/paper/2212.05987/) (2022)
- [Geometric Insights into Focal Loss: Reducing Curvature for Enhanced Model Calibration](https://mlsanity.com/paper/2405.00442/) (2024)
- [Enhancing Financial Market Predictions: Causality-Driven Feature Selection](https://mlsanity.com/paper/2408.01005/) (2024)
- [Towards Understanding The Calibration Benefits of Sharpness-Aware Minimization](https://mlsanity.com/paper/2505.23866/) (2025)
- [Towards Unbiased Calibration using Meta-Regularization](https://mlsanity.com/paper/2303.15057/) (2023)
- [Enhance GNNs with Reliable Confidence Estimation via Adversarial Calibration Learning](https://mlsanity.com/paper/2503.18235/) (2025)
- [Probability Bounding: Post-Hoc Calibration via Box-Constrained Softmax](https://mlsanity.com/paper/2506.10572/) (2025)
- [Deep Neural Network Calibration by Reducing Classifier Shift with Stochastic Masking](https://mlsanity.com/paper/2508.09116/) (2025)
- [FedLAS: Feature-Modulated Bidirectional Label Smoothing for Neural Network Calibration](https://mlsanity.com/paper/2606.28654/) (2026)
- [A Guide to Failure in Machine Learning: Reliability and Robustness from Foundations to Practice](https://mlsanity.com/paper/2503.00563/) (2025)
- [Revisiting Reweighted Risk for Calibration: AURC, Focal, and Inverse Focal Loss](https://mlsanity.com/paper/2505.23463/) (2025)
- [An Enhanced Focal Loss Function to Mitigate Class Imbalance in Auto Insurance Fraud Detection with Explainable AI](https://mlsanity.com/paper/2508.02283/) (2025)
- [DATS: Distance-Aware Temperature Scaling for Calibrated Class-Incremental Learning](https://mlsanity.com/paper/2509.21161/) (2025)
- [Calibrating the Full Predictive Class Distribution of 3D Object Detectors for Autonomous Driving](https://mlsanity.com/paper/2510.01829/) (2025)
- [Learning to Retrieve with Weakened Labels: Robust Training under Label Noise](https://mlsanity.com/paper/2512.13237/) (2025)
- [Deep Probabilistic Supervision for Image Classification](https://mlsanity.com/paper/2512.24162/) (2025)
- [Functional Properties of the Focal-Entropy](https://mlsanity.com/paper/2603.02533/) (2026)
- [Generative Cross-Entropy: A Strictly Proper Loss for Data-Efficient Classification](https://mlsanity.com/paper/2604.06689/) (2026)
- [Socrates Loss: Unifying Confidence Calibration and Classification by Leveraging the Unknown](https://mlsanity.com/paper/2604.12245/) (2026)

## Abstract (arXiv)

Much recent work has been devoted to the problem of ensuring that a neural network's confidence scores match the true probability of being correct, i.e. the calibration problem. Of note, it was found that training with focal loss leads to better calibration than cross-entropy while achieving similar level of accuracy citemukhoti2020. This success stems from focal loss regularizing the entropy of the model's prediction (controlled by the parameter γ), thereby reining in the model's overconfidence. Further improvement is expected if γ is selected independently for each training sample (Sample-Dependent Focal Loss (FLSD-53) citemukhoti2020). However, FLSD-53 is based on heuristics and does not generalize well. In this paper, we propose a calibration-aware adaptive focal loss called AdaFocal that utilizes the calibration properties of focal (and inverse-focal) loss and adaptively modifies γt for different groups of samples based on γt-1 from the previous step and the knowledge of model's under/over-confidence on the validation set. We evaluate AdaFocal on various image recognition and one NLP task, covering a wide variety of network architectures, to confirm the improvement in calibration while achieving similar levels of accuracy. Additionally, we show that models trained with AdaFocal achieve a significant boost in out-of-distribution detection.