# Ensuring Reliability via Hyperparameter Selection: Review and Advances

arXiv: https://arxiv.org/abs/2502.04206 · HTML: https://mlsanity.com/paper/2502.04206/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 6 Feb 2025.

## Builds on (most cited first, 17 of 17)

- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [Confident Adaptive Language Modeling](https://mlsanity.com/paper/2207.07061/) (2022)
- [Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control](https://mlsanity.com/paper/2110.01052/) (2021)
- [On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice](https://mlsanity.com/paper/2007.15745/) (2020)
- [Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging](https://mlsanity.com/paper/2202.05265/) (2022)
- [Hypothesis testing with e-values](https://mlsanity.com/paper/2410.23614/) (2024)
- [Efficiently Controlling Multiple Risks with Pareto Testing](https://mlsanity.com/paper/2210.07913/) (2022)
- [Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selection](https://mlsanity.com/paper/2409.15844/) (2024)
- [Semi-Supervised Risk Control via Prediction-Powered Inference](https://mlsanity.com/paper/2412.11174/) (2024)
- [Online multiple testing with e-values](https://mlsanity.com/paper/2311.06412/) (2023)
- [An Information-Theoretic Approach to Generalization Theory](https://mlsanity.com/paper/2408.13275/) (2024)
- [What, Indeed, is an Achievable Provable Guarantee for Learning-Enabled Safety Critical Systems](https://mlsanity.com/paper/2307.11784/) (2023)
- [Quantile Learn-Then-Test: Quantile-Based Risk Control for Hyperparameter Optimization](https://mlsanity.com/paper/2407.17358/) (2024)
- [What If We Had Used a Different App? Reliable Counterfactual KPI Analysis in Wireless Systems](https://mlsanity.com/paper/2410.00150/) (2024)
- [Learning to Broadcast for Ultra-Reliable Communication with Differential Quality of Service via the Conditional Value at Risk](https://mlsanity.com/paper/2112.02007/) (2021)
- [Statistically Valid Information Bottleneck via Multiple Hypothesis Testing](https://mlsanity.com/paper/2409.07325/) (2024)
- [Multi-Objective Hyperparameter Selection via Hypothesis Testing on Reliability Graphs](https://mlsanity.com/paper/2501.13018/) (2025)

## Built on by (most cited first, 3 of 3)

- [Uncertainty Quantification and Data Efficiency in AI: An Information-Theoretic Perspective](https://mlsanity.com/paper/2512.05267/) (2025)
- [Reliable LLM-Based Edge-Cloud-Expert Cascades for Telecom Knowledge Systems](https://mlsanity.com/paper/2512.20012/) (2025)
- [Post-Selection Distributional Model Evaluation](https://mlsanity.com/paper/2603.23055/) (2026)

## Abstract (arXiv)

Hyperparameter selection is a critical step in the deployment of artificial intelligence (AI) models, particularly in the current era of foundational, pre-trained, models. By framing hyperparameter selection as a multiple hypothesis testing problem, recent research has shown that it is possible to provide statistical guarantees on population risk measures attained by the selected hyperparameter. This paper reviews the Learn-Then-Test (LTT) framework, which formalizes this approach, and explores several extensions tailored to engineering-relevant scenarios. These extensions encompass different risk measures and statistical guarantees, multi-objective optimization, the incorporation of prior knowledge and dependency structures into the hyperparameter selection process, as well as adaptivity. The paper also includes illustrative applications for communication systems.