# Selecting the Best Optimizing System

arXiv: https://arxiv.org/abs/2201.03065 · HTML: https://mlsanity.com/paper/2201.03065/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 56 references that are in the map.
- First posted on arXiv 9 Jan 2022.

## Builds on (most cited first, 9 of 9)

- [Simple Bayesian Algorithms for Best Arm Identification](https://mlsanity.com/paper/1602.08448/) (2016)
- [Tight (Lower) Bounds for the Fixed Budget Best Arm Identification Bandit Problem](https://mlsanity.com/paper/1605.09004/) (2016)
- [Review on Ranking and Selection: A New Perspective](https://mlsanity.com/paper/2008.00249/) (2020)
- [Ranking and Selection with Covariates for Personalized Decision Making](https://mlsanity.com/paper/1710.02642/) (2017)
- [An online learning approach to dynamic pricing and capacity sizing in service systems](https://mlsanity.com/paper/2009.02911/) (2020)
- [Analyzing and provably improving fixed budget ranking and selection algorithms](https://mlsanity.com/paper/1811.12183/) (2018)
- [Distributionally Robust Selection of the Best](https://mlsanity.com/paper/1903.05828/) (2019)
- [Asymptotically Optimal Sampling Policy for Selecting Top-m Alternatives](https://mlsanity.com/paper/2111.15172/) (2021)
- [Context-dependent Ranking and Selection under a Bayesian Framework](https://mlsanity.com/paper/2012.05577/) (2020)

## Abstract (arXiv)

We formulate selecting the best optimizing system (SBOS) problems and provide solutions for those problems. In an SBOS problem, a finite number of systems are contenders. Inside each system, a continuous decision variable affects the system's expected performance. An SBOS problem compares different systems based on their expected performances under their own optimally chosen decision to select the best, without advance knowledge of expected performances of the systems nor the optimizing decision inside each system. We design easy-to-implement algorithms that adaptively chooses a system and a choice of decision to evaluate the noisy system performance, sequentially eliminates inferior systems, and eventually recommends a system as the best after spending a user-specified budget. The proposed algorithms integrate the stochastic gradient descent method and the sequential elimination method to simultaneously exploit the structure inside each system and make comparisons across systems. For the proposed algorithms, we prove exponential rates of convergence to zero for the probability of false selection, as the budget grows to infinity. We conduct three numerical examples that represent three practical cases of SBOS problems. Our proposed algorithms demonstrate consistent and stronger performances in terms of the probability of false selection over benchmark algorithms under a range of problem settings and sampling budgets.