# Quantile Learn-Then-Test: Quantile-Based Risk Control for Hyperparameter Optimization

arXiv: https://arxiv.org/abs/2407.17358 · HTML: https://mlsanity.com/paper/2407.17358/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 22 references that are in the map.
- First posted on arXiv 24 Jul 2024.

## Builds on (most cited first, 10 of 10)

- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Confident Adaptive Language Modeling](https://mlsanity.com/paper/2207.07061/) (2022)
- [Distribution-Free, Risk-Controlling Prediction Sets](https://mlsanity.com/paper/2101.02703/) (2021)
- [Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control](https://mlsanity.com/paper/2110.01052/) (2021)
- [The Roadmap to 6G -- AI Empowered Wireless Networks](https://mlsanity.com/paper/1904.11686/) (2019)
- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [Improving Trustworthiness of AI Disease Severity Rating in Medical Imaging with Ordinal Conformal Prediction Sets](https://mlsanity.com/paper/2207.02238/) (2022)
- [Sequential estimation of quantiles with applications to A/B-testing and best-arm identification](https://mlsanity.com/paper/1906.09712/) (2019)
- [Quantile Risk Control: A Flexible Framework for Bounding the Probability of High-Loss Predictions](https://mlsanity.com/paper/2212.13629/) (2022)
- [Distribution-Free Statistical Dispersion Control for Societal Applications](https://mlsanity.com/paper/2309.13786/) (2023)

## Built on by (most cited first, 6 of 6)

- [Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selection](https://mlsanity.com/paper/2409.15844/) (2024)
- [Conformal Prediction as Bayesian Quadrature](https://mlsanity.com/paper/2502.13228/) (2025)
- [Ensuring Reliability via Hyperparameter Selection: Review and Advances](https://mlsanity.com/paper/2502.04206/) (2025)
- [Reliable LLM-Based Edge-Cloud-Expert Cascades for Telecom Knowledge Systems](https://mlsanity.com/paper/2512.20012/) (2025)
- [Multi-Objective Hyperparameter Selection via Hypothesis Testing on Reliability Graphs](https://mlsanity.com/paper/2501.13018/) (2025)
- [Occupancy-based Quantile Risk Control](https://mlsanity.com/paper/2609.03104/) (2026)

## Abstract (arXiv)

The increasing adoption of Artificial Intelligence (AI) in engineering problems calls for the development of calibration methods capable of offering robust statistical reliability guarantees. The calibration of black box AI models is carried out via the optimization of hyperparameters dictating architecture, optimization, and/or inference configuration. Prior work has introduced learn-then-test (LTT), a calibration procedure for hyperparameter optimization (HPO) that provides statistical guarantees on average performance measures. Recognizing the importance of controlling risk-aware objectives in engineering contexts, this work introduces a variant of LTT that is designed to provide statistical guarantees on quantiles of a risk measure. We illustrate the practical advantages of this approach by applying the proposed algorithm to a radio access scheduling problem.