# Calibrate: Interactive Analysis of Probabilistic Model Output

arXiv: https://arxiv.org/abs/2207.13770 · HTML: https://mlsanity.com/paper/2207.13770/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 51 references that are in the map.
- First posted on arXiv 27 Jul 2022.

## Builds on (most cited first, 15 of 15)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [InterpretML: A Unified Framework for Machine Learning Interpretability](https://mlsanity.com/paper/1909.09223/) (2019)
- [Evaluating model calibration in classification](https://mlsanity.com/paper/1902.06977/) (2019)
- [Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers](https://mlsanity.com/paper/1801.06889/) (2018)
- [FairVis: Visual Analytics for Discovering Intersectional Bias in Machine Learning](https://mlsanity.com/paper/1904.05419/) (2019)
- [Calibration tests in multi-class classification: A unifying framework](https://mlsanity.com/paper/1910.11385/) (2019)
- [ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models](https://mlsanity.com/paper/1704.01942/) (2017)
- [Boxer: Interactive Comparison of Classifier Results](https://mlsanity.com/paper/2004.07964/) (2020)
- [Evaluating probabilistic classifiers: Reliability diagrams and score decompositions revisited](https://mlsanity.com/paper/2008.03033/) (2020)
- [Neo: Generalizing Confusion Matrix Visualization to Hierarchical and Multi-Output Labels](https://mlsanity.com/paper/2110.12536/) (2021)
- [ConfusionFlow: A model-agnostic visualization for temporal analysis of classifier confusion](https://mlsanity.com/paper/1910.00969/) (2019)
- [Histogram binning revisited with a focus on human perception](https://mlsanity.com/paper/2109.06612/) (2021)

## Built on by (most cited first, 6 of 6)

- [Visual Analytics For Machine Learning: A Data Perspective Survey](https://mlsanity.com/paper/2307.07712/) (2023)
- [Evaluating Probabilistic Classifiers: The Triptych](https://mlsanity.com/paper/2301.10803/) (2023)
- [A Unified Interactive Model Evaluation for Classification, Object Detection, and Instance Segmentation in Computer Vision](https://mlsanity.com/paper/2308.05168/) (2023)
- [SuperNOVA: Design Strategies and Opportunities for Interactive Visualization in Computational Notebooks](https://mlsanity.com/paper/2305.03039/) (2023)
- [Data-Driven Prediction and Uncertainty Quantification of PWR Crud-Induced Power Shift Using Convolutional Neural Networks](https://mlsanity.com/paper/2407.04726/) (2024)
- [SimClinician: A Multimodal Simulation Testbed for Reliable Psychologist AI Collaboration in Mental Health Diagnosis](https://mlsanity.com/paper/2512.08953/) (2025)

## Abstract (arXiv)

Analyzing classification model performance is a crucial task for machine learning practitioners. While practitioners often use count-based metrics derived from confusion matrices, like accuracy, many applications, such as weather prediction, sports betting, or patient risk prediction, rely on a classifier's predicted probabilities rather than predicted labels. In these instances, practitioners are concerned with producing a calibrated model, that is, one which outputs probabilities that reflect those of the true distribution. Model calibration is often analyzed visually, through static reliability diagrams, however, the traditional calibration visualization may suffer from a variety of drawbacks due to the strong aggregations it necessitates. Furthermore, count-based approaches are unable to sufficiently analyze model calibration. We present Calibrate, an interactive reliability diagram that addresses the aforementioned issues. Calibrate constructs a reliability diagram that is resistant to drawbacks in traditional approaches, and allows for interactive subgroup analysis and instance-level inspection. We demonstrate the utility of Calibrate through use cases on both real-world and synthetic data. We further validate Calibrate by presenting the results of a think-aloud experiment with data scientists who routinely analyze model calibration.