# Learning to Assess the Reliability of Number-of-Runs Estimation in Stochastic Optimization

arXiv: https://arxiv.org/abs/2605.28309 · HTML: https://mlsanity.com/paper/2605.28309/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 17 references that are in the map.
- First posted on arXiv 27 May 2026.

## Builds on (most cited first, 6 of 6)

- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting](https://mlsanity.com/paper/1603.08785/) (2016)
- [Benchmarking in Optimization: Best Practice and Open Issues](https://mlsanity.com/paper/2007.03488/) (2020)
- [Analyzing the Impact of Undersampling on the Benchmarking and Configuration of Evolutionary Algorithms](https://mlsanity.com/paper/2204.09353/) (2022)
- [Adaptive Estimation of the Number of Algorithm Runs in Stochastic Optimization](https://mlsanity.com/paper/2507.01629/) (2025)
- [Sample size estimation for power and accuracy in the experimental comparison of algorithms](https://mlsanity.com/paper/1808.02997/) (2018)

## Abstract (arXiv)

In large-scale benchmarking of stochastic optimization algorithms, the key challenge is no longer whether repeated runs are needed for reliability, but how to determine when sufficient evidence has been collected without incurring unnecessary computational cost. We study a learning-based extension of a recent empirical online heuristic that adaptively estimates the required number of runs using outlier handling and skewness-based symmetry checks. Using annotated outcomes from 132{,}000 Nevergrad runs on COCO (24 problems in 20 dimensions, 10 instances each, 11 optimizers), we train classifiers on 23 statistical, energy-free, and shape and stability features to predict whether a run-number estimate is reliable, prioritizing detection of incorrect estimates via minority-class recall. We evaluate reliability prediction using a within-configuration learning setup, where models are trained and tested on data sharing the same optimizer. The results show that run-number reliability can be learned in a within-configuration scenario, enabling detection of unreliable estimates with high minority-class recall, although performance remains limited by the restricted data diversity within fixed configurations.