# Good practices for evaluation of machine learning systems

arXiv: https://arxiv.org/abs/2412.03700 · HTML: https://mlsanity.com/paper/2412.03700/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 13 references that are in the map.
- First posted on arXiv 4 Dec 2024.

## Builds on (most cited first, 4 of 4)

- [Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning](https://mlsanity.com/paper/1811.12808/) (2018)
- [Evaluating Posterior Probabilities: Decision Theory, Proper Scoring Rules, and Calibration](https://mlsanity.com/paper/2408.02841/) (2024)
- [The Unreliability of Acoustic Systems in Alzheimer's Speech Datasets with Heterogeneous Recording Conditions](https://mlsanity.com/paper/2409.12170/) (2024)
- [Does the evaluation stand up to evaluation? A first-principle approach to the evaluation of classifiers](https://mlsanity.com/paper/2302.12006/) (2023)

## Built on by (most cited first, 5 of 5)

- [Mitigating Subgroup Disparities in Multi-Label Speech Emotion Recognition: A Pseudo-Labeling and Unsupervised Learning Approach](https://mlsanity.com/paper/2505.14449/) (2025)
- [Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting](https://mlsanity.com/paper/2606.09809/) (2026)
- [Does language matter for spoken word classification? A multilingual generative meta-learning approach](https://mlsanity.com/paper/2605.13084/) (2026)
- [Comparing Human and Automatic Recognition of Dutch Dysarthric Continuous Speech: A Case Study](https://mlsanity.com/paper/2606.30237/) (2026)
- [Benchmarking Human and Automatic Speech Recognition of Diverse Speech: Initial Results](https://mlsanity.com/paper/2607.19049/) (2026)

## Abstract (arXiv)

Many development decisions affect the results obtained from ML experiments: training data, features, model architecture, hyperparameters, test data, etc. Among these aspects, arguably the most important design decisions are those that involve the evaluation procedure. This procedure is what determines whether the conclusions drawn from the experiments will or will not generalize to unseen data and whether they will be relevant to the application of interest. If the data is incorrectly selected, the wrong metric is chosen for evaluation or the significance of the comparisons between models is overestimated, conclusions may be misleading or result in suboptimal development decisions. To avoid such problems, the evaluation protocol should be very carefully designed before experimentation starts. In this work we discuss the main aspects involved in the design of the evaluation protocol: data selection, metric selection, and statistical significance. This document is not meant to be an exhaustive tutorial on each of these aspects. Instead, the goal is to explain the main guidelines that should be followed in each case. We include examples taken from the speech processing field, and provide a list of common mistakes related to each aspect.