# Master your Metrics with Calibration

arXiv: https://arxiv.org/abs/1909.02827 · HTML: https://mlsanity.com/paper/1909.02827/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Sep 2019.

## Built on by (most cited first, 8 of 8)

- [The Importance of Future Information in Credit Card Fraud Detection](https://mlsanity.com/paper/2204.05265/) (2022)
- [ImDrug: A Benchmark for Deep Imbalanced Learning in AI-aided Drug Discovery](https://mlsanity.com/paper/2209.07921/) (2022)
- [COPA: Comparing the incomparable in multi-objective model evaluation](https://mlsanity.com/paper/2503.14321/) (2025)
- [Steering LLMs? Actually, Sparse Autoencoders can outperform simple baselines](https://mlsanity.com/paper/2605.31183/) (2026)
- [End-To-End Prediction of Knee Osteoarthritis Progression With Multi-Modal Transformers](https://mlsanity.com/paper/2307.00873/) (2023)
- [Transfer Learning for Credit Card Fraud Detection: A Journey from Research to Production](https://mlsanity.com/paper/2107.09323/) (2021)
- [HOLMES: HOLonym-MEronym based Semantic inspection for Convolutional Image Classifiers](https://mlsanity.com/paper/2403.08536/) (2024)
- [A Systematic Evaluation of Imbalance Handling Methods in Biomedical Binary Classification](https://mlsanity.com/paper/2605.14147/) (2026)

## Abstract (arXiv)

Machine learning models deployed in real-world applications are often evaluated with precision-based metrics such as F1-score or AUC-PR (Area Under the Curve of Precision Recall). Heavily dependent on the class prior, such metrics make it difficult to interpret the variation of a model's performance over different subpopulations/subperiods in a dataset. In this paper, we propose a way to calibrate the metrics so that they can be made invariant to the prior. We conduct a large number of experiments on balanced and imbalanced data to assess the behavior of calibrated metrics and show that they improve interpretability and provide a better control over what is really measured. We describe specific real-world use-cases where calibration is beneficial such as, for instance, model monitoring in production, reporting, or fairness evaluation.