# Futility Analysis in the Cross-Validation of Machine Learning Models

arXiv: https://arxiv.org/abs/1405.6974 · HTML: https://mlsanity.com/paper/1405.6974/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2014.

## Built on by (most cited first, 4 of 4)

- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [Machine learning and economic forecasting: the role of international trade networks](https://mlsanity.com/paper/2404.08712/) (2024)
- [Leveraging advances in machine learning for the robust classification and interpretation of networks](https://mlsanity.com/paper/2403.13215/) (2024)
- [A Comparison of Deep Learning and Established Methods for Calf Behaviour Monitoring](https://mlsanity.com/paper/2408.13041/) (2024)

## Abstract (arXiv)

Many machine learning models have important structural tuning parameters that cannot be directly estimated from the data. The common tactic for setting these parameters is to use resampling methods, such as cross--validation or the bootstrap, to evaluate a candidate set of values and choose the best based on some pre--defined criterion. Unfortunately, this process can be time consuming. However, the model tuning process can be streamlined by adaptively resampling candidate values so that settings that are clearly sub-optimal can be discarded. The notion of futility analysis is introduced in this context. An example is shown that illustrates how adaptive resampling can be used to reduce training time. Simulation studies are used to understand how the potential speed--up is affected by parallel processing techniques.