# Have we been Naive to Select Machine Learning Models? Noisy Data are here to Stay!

arXiv: https://arxiv.org/abs/2207.06651 · HTML: https://mlsanity.com/paper/2207.06651/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 25 references that are in the map.
- First posted on arXiv 14 Jul 2022.

## Builds on (most cited first, 5 of 5)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Similarity Based Stratified Splitting: an approach to train better classifiers](https://mlsanity.com/paper/2010.06099/) (2020)
- [Embarrassingly Parallel Independent Training of Multi-Layer Perceptrons with Heterogeneous Architectures](https://mlsanity.com/paper/2206.08369/) (2022)

## Built on by (most cited first, 1 of 1)

- [A Lightweight and Extensible Cell Segmentation and Classification Model for Whole Slide Images](https://mlsanity.com/paper/2502.19217/) (2025)

## Abstract (arXiv)

The model selection procedure is usually a single-criterion decision making in which we select the model that maximizes a specific metric in a specific set, such as the Validation set performance. We claim this is very naive and can perform poor selections of over-fitted models due to the over-searching phenomenon, which over-estimates the performance on that specific set. Futhermore, real world data contains noise that should not be ignored by the model selection procedure and must be taken into account when performing model selection. Also, we have defined four theoretical optimality conditions that we can pursue to better select the models and analyze them by using a multi-criteria decision-making algorithm (TOPSIS) that considers proxies to the optimality conditions to select reasonable models.