# Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning

arXiv: https://arxiv.org/abs/2003.07329 · HTML: https://mlsanity.com/paper/2003.07329/

## Key facts

- Cited by 190 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Mar 2020.

## Built on by (most cited first, 30 of 190)

- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Taming Overconfidence in LLMs: Reward Calibration in RLHF](https://mlsanity.com/paper/2410.09724/) (2024)
- [Be Confident! Towards Trustworthy Graph Neural Networks via Confidence Calibration](https://mlsanity.com/paper/2109.14285/) (2021)
- [The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration](https://mlsanity.com/paper/2111.15430/) (2021)
- [Uncertainty Quantification over Graph with Conformalized Graph Neural Networks](https://mlsanity.com/paper/2305.14535/) (2023)
- [Calibration in Deep Learning: A Survey of the State-of-the-Art](https://mlsanity.com/paper/2308.01222/) (2023)
- [Efficient Test-Time Scaling via Self-Calibration](https://mlsanity.com/paper/2503.00031/) (2025)
- [Classifier Calibration: A survey on how to assess and improve predicted class probabilities](https://mlsanity.com/paper/2112.10327/) (2021)
- [Better Uncertainty Calibration via Proper Scores for Classification and Beyond](https://mlsanity.com/paper/2203.07835/) (2022)
- [A Unifying Theory of Distance from Calibration](https://mlsanity.com/paper/2211.16886/) (2022)
- [Parameterized Temperature Scaling for Boosting the Expressive Power in Post-Hoc Uncertainty Calibration](https://mlsanity.com/paper/2102.12182/) (2021)
- [A Consistent and Differentiable Lp Canonical Calibration Error Estimator](https://mlsanity.com/paper/2210.07810/) (2022)
- [Top-label calibration and multiclass-to-binary reductions](https://mlsanity.com/paper/2107.08353/) (2021)
- [LitCab: Lightweight Language Model Calibration over Short- and Long-form Responses](https://mlsanity.com/paper/2310.19208/) (2023)
- [Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibration](https://mlsanity.com/paper/2109.14591/) (2021)
- [Metrics for saliency map evaluation of deep learning explanation methods](https://mlsanity.com/paper/2201.13291/) (2022)
- [What Makes Graph Neural Networks Miscalibrated?](https://mlsanity.com/paper/2210.06391/) (2022)
- [Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification](https://mlsanity.com/paper/2102.07856/) (2021)
- [ConfTuner: Training Large Language Models to Express Their Confidence Verbally](https://mlsanity.com/paper/2508.18847/) (2025)
- [Towards Calibrated Robust Fine-Tuning of Vision-Language Models](https://mlsanity.com/paper/2311.01723/) (2023)
- [Meta-Cal: Well-controlled Post-hoc Calibration by Ranking](https://mlsanity.com/paper/2105.04290/) (2021)
- [Dual Focal Loss for Calibration](https://mlsanity.com/paper/2305.13665/) (2023)
- [MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks](https://mlsanity.com/paper/2103.06132/) (2021)
- [AdaFocal: Calibration-aware Adaptive Focal Loss](https://mlsanity.com/paper/2211.11838/) (2022)
- [A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainability](https://mlsanity.com/paper/2212.10888/) (2022)
- [Confidence Calibration for Domain Generalization under Covariate Shift](https://mlsanity.com/paper/2104.00742/) (2021)
- [Don't Go To Extremes: Revealing the Excessive Sensitivity and Calibration Limitations of LLMs in Implicit Hate Speech Detection](https://mlsanity.com/paper/2402.11406/) (2024)
- [Local Calibration: Metrics and Recalibration](https://mlsanity.com/paper/2102.10809/) (2021)
- [T-Cal: An optimal test for the calibration of predictive models](https://mlsanity.com/paper/2203.01850/) (2022)

## Abstract (arXiv)

This paper studies the problem of post-hoc calibration of machine learning classifiers. We introduce the following desiderata for uncertainty calibration: (a) accuracy-preserving, (b) data-efficient, and (c) high expressive power. We show that none of the existing methods satisfy all three requirements, and demonstrate how Mix-n-Match calibration strategies (i.e., ensemble and composition) can help achieve remarkably better data-efficiency and expressive power while provably maintaining the classification accuracy of the original classifier. Mix-n-Match strategies are generic in the sense that they can be used to improve the performance of any off-the-shelf calibrator. We also reveal potential issues in standard evaluation practices. Popular approaches (e.g., histogram-based expected calibration error (ECE)) may provide misleading results especially in small-data regime. Therefore, we propose an alternative data-efficient kernel density-based estimator for a reliable evaluation of the calibration performance and prove its asymptotically unbiasedness and consistency. Our approaches outperform state-of-the-art solutions on both the calibration as well as the evaluation tasks in most of the experimental settings. Our codes are available at https://github.com/zhang64-llnl/Mix-n-Match-Calibration.