# The use of cross validation in the analysis of designed experiments

arXiv: https://arxiv.org/abs/2506.14593 · HTML: https://mlsanity.com/paper/2506.14593/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 75 references that are in the map.
- First posted on arXiv 17 Jun 2025.

## Builds on (most cited first, 8 of 8)

- [ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R](https://mlsanity.com/paper/1508.04409/) (2015)
- [Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning](https://mlsanity.com/paper/1811.12808/) (2018)
- [Tunability: Importance of Hyperparameters of Machine Learning Algorithms](https://mlsanity.com/paper/1802.09596/) (2018)
- [Cross-validation: what does it estimate and how well does it do it?](https://mlsanity.com/paper/2104.00673/) (2021)
- [Assessing and tuning brain decoders: cross-validation, caveats, and guidelines](https://mlsanity.com/paper/1606.05201/) (2016)
- [Is K-fold cross validation the best model selection method for Machine Learning?](https://mlsanity.com/paper/2401.16407/) (2024)
- [Self-Validated Ensemble Models for Design of Experiments](https://mlsanity.com/paper/2103.09303/) (2021)
- [Design choice and machine learning model performances](https://mlsanity.com/paper/2201.10239/) (2022)

## Abstract (arXiv)

Cross-validation (CV) is a common method to tune machine learning methods and can be used for model selection in regression as well. Because of the structured nature of small, traditional experimental designs, the literature has warned against using CV in their analysis. The striking increase in the use of machine learning, and thus CV, in the analysis of experimental designs, has led us to empirically study the effectiveness of CV compared to other methods of selecting models in designed experiments, including the little bootstrap. We consider both response surface settings where prediction is of primary interest, as well as screening where factor selection is most important. Overall, we provide evidence that the use of leave-one-out cross-validation (LOOCV) in the analysis of small, structured is often useful. More general k-fold CV may also be competitive but its performance is uneven.