# Quantile-Scaled Bayesian Optimization Using Rank-Only Feedback

arXiv: https://arxiv.org/abs/2510.03277 · HTML: https://mlsanity.com/paper/2510.03277/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 22 references that are in the map.
- First posted on arXiv 28 Sep 2025.

## Builds on (most cited first, 9 of 9)

- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [A System for Massively Parallel Hyperparameter Tuning](https://mlsanity.com/paper/1810.05934/) (2018)
- [Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance](https://mlsanity.com/paper/2304.11127/) (2023)
- [Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets](https://mlsanity.com/paper/1605.07079/) (2016)
- [Recent Advances in Bayesian Optimization](https://mlsanity.com/paper/2206.03301/) (2022)
- [Top-k Ranking Bayesian Optimization](https://mlsanity.com/paper/2012.10688/) (2020)
- [Achieving Robustness to Aleatoric Uncertainty with Heteroscedastic Bayesian Optimisation](https://mlsanity.com/paper/1910.07779/) (2019)
- [Ranking over Regression for Bayesian Optimization and Molecule Selection](https://mlsanity.com/paper/2410.09290/) (2024)

## Abstract (arXiv)

Bayesian Optimization (BO) is widely used for optimizing expensive black-box functions, particularly in hyperparameter tuning. However, standard BO assumes access to precise objective values, which may be unavailable, noisy, or unreliable in real-world settings where only relative or rank-based feedback can be obtained. In this study, we propose Quantile-Scaled Bayesian Optimization (QS-BO), a principled rank-based optimization framework. QS-BO converts ranks into heteroscedastic Gaussian targets through a quantile-scaling pipeline, enabling the use of Gaussian process surrogates and standard acquisition functions without requiring explicit metric scores. We evaluate QS-BO on synthetic benchmark functions, including one- and two-dimensional nonlinear functions and the Branin function, and compare its performance against Random Search. Results demonstrate that QS-BO consistently achieves lower objective values and exhibits greater stability across runs. Statistical tests further confirm that QS-BO significantly outperforms Random Search at the 1% significance level. These findings establish QS-BO as a practical and effective extension of Bayesian Optimization for rank-only feedback, with promising applications in preference learning, recommendation, and human-in-the-loop optimization where absolute metric values are unavailable or unreliable.