# Monitoring the calibration of probability forecasts with an application to concept drift detection involving image classification

arXiv: https://arxiv.org/abs/2510.25573 · HTML: https://mlsanity.com/paper/2510.25573/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 30 references that are in the map.
- First posted on arXiv 29 Oct 2025.

## Builds on (most cited first, 4 of 4)

- [Classifier Calibration: A survey on how to assess and improve predicted class probabilities](https://mlsanity.com/paper/2112.10327/) (2021)
- [Concept Drift Monitoring and Diagnostics of Supervised Learning Models via Score Vectors](https://mlsanity.com/paper/2012.06916/) (2020)
- [Boldness-Recalibration for Binary Event Predictions](https://mlsanity.com/paper/2305.03780/) (2023)
- [BRcal: An R Package to Boldness-Recalibrate Probability Predictions](https://mlsanity.com/paper/2409.13858/) (2024)

## Abstract (arXiv)

Machine learning approaches for image classification have led to impressive advances in that field. For example, convolutional neural networks are able to achieve remarkable image classification accuracy across a wide range of applications in industry, defense, and other areas. While these machine learning models boast impressive accuracy, a related concern is how to assess and maintain calibration in the predictions these models make. A classification model is said to be well calibrated if its predicted probabilities correspond with the rates events actually occur. While there are many available methods to assess machine learning calibration and recalibrate faulty predictions, less effort has been spent on developing approaches that continually monitor predictive models for potential loss of calibration as time passes. We propose a cumulative sum-based approach with dynamic limits that enable detection of miscalibration in both traditional process monitoring and concept drift applications. This enables early detection of operational context changes that impact image classification performance in the field. The proposed chart can be used broadly in any situation where the user needs to monitor probability predictions over time for potential lapses in calibration. Importantly, our method operates on probability predictions and event outcomes and does not require under-the-hood access to the machine learning model.