# Variation in prediction accuracy due to randomness in data division and fair evaluation using interval estimation

arXiv: https://arxiv.org/abs/2409.01025 · HTML: https://mlsanity.com/paper/2409.01025/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 1 of its 20 references that are in the map.
- First posted on arXiv 2 Sep 2024.

## Builds on (most cited first, 1 of 1)

- [Classification of Diabetes Mellitus using Modified Particle Swarm Optimization and Least Squares Support Vector Machine](https://mlsanity.com/paper/1405.0549/) (2014)

## Abstract (arXiv)

This paper attempts to answer a "simple question" in building predictive models using machine learning algorithms. Although diagnostic and predictive models for various diseases have been proposed using data from large cohort studies and machine learning algorithms, challenges remain in their generalizability. Several causes for this challenge have been pointed out, and partitioning of the dataset with randomness is considered to be one of them. In this study, we constructed 33,600 diabetes diagnosis models with "initial state" dependent randomness using autoML (automatic machine learning framework) and open diabetes data, and evaluated their prediction accuracy. The results showed that the prediction accuracy had an initial state-dependent distribution. Since this distribution could follow a normal distribution, we estimated the expected interval of prediction accuracy using statistical interval estimation in order to fairly compare the accuracy of the prediction models.