# Dual Focal Loss for Calibration

arXiv: https://arxiv.org/abs/2305.13665 · HTML: https://mlsanity.com/paper/2305.13665/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 56 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks](https://mlsanity.com/paper/1905.11001/) (2019)
- [Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning](https://mlsanity.com/paper/2003.07329/) (2020)
- [Uncertainty Quantification and Deep Ensembles](https://mlsanity.com/paper/2007.08792/) (2020)
- [Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks](https://mlsanity.com/paper/2006.07322/) (2020)
- [Improving model calibration with accuracy versus uncertainty optimization](https://mlsanity.com/paper/2012.07923/) (2020)
- [Mitigating Bias in Calibration Error Estimation](https://mlsanity.com/paper/2012.08668/) (2020)
- [Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations](https://mlsanity.com/paper/1807.01697/) (2018)
- [Calibration of Neural Networks using Splines](https://mlsanity.com/paper/2006.12800/) (2020)
- [Soft Calibration Objectives for Neural Networks](https://mlsanity.com/paper/2108.00106/) (2021)
- [Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks](https://mlsanity.com/paper/1312.6082/) (2013)
- [Posterior calibration and exploratory analysis for natural language processing models](https://mlsanity.com/paper/1508.05154/) (2015)

## Built on by (most cited first, 30 of 33)

- [A Short Survey on Importance Weighting for Machine Learning](https://mlsanity.com/paper/2403.10175/) (2024)
- [A Benchmark Study on Calibration](https://mlsanity.com/paper/2308.11838/) (2023)
- [Epistemic Artificial Intelligence is Essential for Machine Learning Models to Truly 'Know When They Do Not Know'](https://mlsanity.com/paper/2505.04950/) (2025)
- [Selective Learning: Towards Robust Calibration with Dynamic Regularization](https://mlsanity.com/paper/2402.08384/) (2024)
- [Uncertainty Weighted Gradients for Model Calibration](https://mlsanity.com/paper/2503.22725/) (2025)
- [Variational Supervised Contrastive Learning](https://mlsanity.com/paper/2506.07413/) (2025)
- [Optimizing Calibration by Gaining Aware of Prediction Correctness](https://mlsanity.com/paper/2404.13016/) (2024)
- [LumiNet: Perception-Driven Knowledge Distillation via Statistical Logit Calibration](https://mlsanity.com/paper/2310.03669/) (2023)
- [Consistency Calibration: Improving Uncertainty Calibration via Consistency among Perturbed Neighbors](https://mlsanity.com/paper/2410.12295/) (2024)
- [Balancing Two Classifiers via A Simplex ETF Structure for Model Calibration](https://mlsanity.com/paper/2504.10007/) (2025)
- [Sample Margin-Aware Recalibration of Temperature Scaling](https://mlsanity.com/paper/2506.23492/) (2025)
- [Geometric Insights into Focal Loss: Reducing Curvature for Enhanced Model Calibration](https://mlsanity.com/paper/2405.00442/) (2024)
- [Enhancing Financial Market Predictions: Causality-Driven Feature Selection](https://mlsanity.com/paper/2408.01005/) (2024)
- [Beyond One-Hot Labels: Semantic Mixing for Model Calibration](https://mlsanity.com/paper/2504.13548/) (2025)
- [Towards Understanding The Calibration Benefits of Sharpness-Aware Minimization](https://mlsanity.com/paper/2505.23866/) (2025)
- [h-calibration: Rethinking Classifier Recalibration with Probabilistic Error-Bounded Objective](https://mlsanity.com/paper/2506.17968/) (2025)
- [Robust Time Series Forecasting with Non-Heavy-Tailed Gaussian Loss-Weighted Sampler](https://mlsanity.com/paper/2406.13871/) (2024)
- [Enhance GNNs with Reliable Confidence Estimation via Adversarial Calibration Learning](https://mlsanity.com/paper/2503.18235/) (2025)
- [Probability Bounding: Post-Hoc Calibration via Box-Constrained Softmax](https://mlsanity.com/paper/2506.10572/) (2025)
- [Instance-Wise Monotonic Calibration by Constrained Transformation](https://mlsanity.com/paper/2507.06516/) (2025)
- [Deep Neural Network Calibration by Reducing Classifier Shift with Stochastic Masking](https://mlsanity.com/paper/2508.09116/) (2025)
- [Target-Agnostic Calibration under Distribution Shift with Frequency-Aware Gradient Rectification](https://mlsanity.com/paper/2508.19830/) (2025)
- [Calibratable Disambiguation Loss for Multi-Instance Partial-Label Learning](https://mlsanity.com/paper/2512.17788/) (2025)
- [FedLAS: Feature-Modulated Bidirectional Label Smoothing for Neural Network Calibration](https://mlsanity.com/paper/2606.28654/) (2026)
- [A Guide to Failure in Machine Learning: Reliability and Robustness from Foundations to Practice](https://mlsanity.com/paper/2503.00563/) (2025)
- [Adaptive Wavelet Filters as Practical Texture Feature Amplifiers for Parkinson's Disease Screening in OCT](https://mlsanity.com/paper/2503.19292/) (2025)
- [Reducing Class Bias In Data-Balanced Datasets Through Hardness-Based Resampling](https://mlsanity.com/paper/2504.07031/) (2025)
- [Revisiting Reweighted Risk for Calibration: AURC, Focal, and Inverse Focal Loss](https://mlsanity.com/paper/2505.23463/) (2025)
- [Calibration Attention: Learning Reliability-Aware Representations for Vision Transformers](https://mlsanity.com/paper/2508.08547/) (2025)
- [A smart fridge with AI-enabled food computing](https://mlsanity.com/paper/2509.07400/) (2025)

## Abstract (arXiv)

The use of deep neural networks in real-world applications require well-calibrated networks with confidence scores that accurately reflect the actual probability. However, it has been found that these networks often provide over-confident predictions, which leads to poor calibration. Recent efforts have sought to address this issue by focal loss to reduce over-confidence, but this approach can also lead to under-confident predictions. While different variants of focal loss have been explored, it is difficult to find a balance between over-confidence and under-confidence. In our work, we propose a new loss function by focusing on dual logits. Our method not only considers the ground truth logit, but also take into account the highest logit ranked after the ground truth logit. By maximizing the gap between these two logits, our proposed dual focal loss can achieve a better balance between over-confidence and under-confidence. We provide theoretical evidence to support our approach and demonstrate its effectiveness through evaluations on multiple models and datasets, where it achieves state-of-the-art performance. Code is available at https://github.com/Linwei94/DualFocalLoss