# A metrological framework for uncertainty evaluation in machine learning classification models

arXiv: https://arxiv.org/abs/2504.03359 · HTML: https://mlsanity.com/paper/2504.03359/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 89 references that are in the map.
- First posted on arXiv 4 Apr 2025.

## Builds on (most cited first, 12 of 12)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Bag of Tricks for Image Classification with Convolutional Neural Networks](https://mlsanity.com/paper/1812.01187/) (2018)
- [Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users](https://mlsanity.com/paper/2007.06823/) (2020)
- [Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL](https://mlsanity.com/paper/2004.13701/) (2020)
- [Analysis and Comparison of Classification Metrics](https://mlsanity.com/paper/2209.05355/) (2022)
- [Uncertainty Quantification for Bayesian Optimization](https://mlsanity.com/paper/2002.01569/) (2020)
- [Robustness of convolutional neural networks to physiological ECG noise](https://mlsanity.com/paper/2108.01995/) (2021)
- [Continental-scale land cover mapping at 10 m resolution over Europe (ELC10)](https://mlsanity.com/paper/2104.10922/) (2021)
- [Aleatoric uncertainty for Errors-in-Variables models in deep regression](https://mlsanity.com/paper/2105.09095/) (2021)
- [An uncertainty-aware Bayesian framework for machine learning classification models: A case study in land cover classification](https://mlsanity.com/paper/2503.21510/) (2025)

## Built on by (most cited first, 4 of 4)

- [Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effects](https://mlsanity.com/paper/2505.18893/) (2025)
- [A systematic evaluation of uncertainty quantification techniques in deep learning: a case study in photoplethysmography signal analysis](https://mlsanity.com/paper/2511.00301/) (2025)
- [Extending confidence calibration to generalised measures of variation](https://mlsanity.com/paper/2602.12975/) (2026)
- [TRACE: A Metrologically-Grounded Engineering Framework for Trustworthy Agentic AI Systems in Operationally Critical Domains](https://mlsanity.com/paper/2605.03838/) (2026)

## Abstract (arXiv)

Machine learning (ML) classification models are increasingly being used in a wide range of applications where it is important that predictions are accompanied by uncertainties, including in climate and earth observation, medical diagnosis and bioaerosol monitoring. The output of an ML classification model is a type of categorical variable known as a nominal property in the International Vocabulary of Metrology (VIM). However, concepts related to uncertainty evaluation for nominal properties are not defined in the VIM, nor is such evaluation addressed by the Guide to the Expression of Uncertainty in Measurement (GUM). In this paper we propose a metrological conceptual uncertainty evaluation framework for nominal properties. This framework is based on probability mass functions and summary statistics thereof, and it is applicable to ML classification. We also illustrate its use in the context of two applications that exemplify the issues and have significant societal impact, namely, climate and earth observation and medical diagnosis. Our framework would enable an extension of the GUM to uncertainty for nominal properties, which would make both applicable to ML classification models.