mlsanityData to 7 Oct 2026

Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selection

arXiv 2409.15844 ↗First posted 24 Sep 2024stat.ML cs.AI cs.ITcited by 13

Not on the map. Why?

Abstract (arXiv)

We introduce adaptive learn-then-test (aLTT), an efficient hyperparameter selection procedure that provides finite-sample statistical guarantees on the population risk of AI models. Unlike the existing learn-then-test (LTT) technique, which relies on conventional p-value-based multiple hypothesis testing (MHT), aLTT implements sequential data-dependent MHT with early termination by leveraging e-processes. As a result, aLTT can reduce the number of testing rounds, making it particularly well-suited for scenarios in which testing is costly or presents safety risks. Apart from maintaining statistical validity, in applications such as online policy selection for offline reinforcement learning and prompt engineering, aLTT is shown to achieve the same performance as LTT while requiring only a fraction of the testing rounds.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 41 of its 74 references that are in the map. First posted on arXiv 24 Sep 2024.

Builds on

41 of 74 references.

References · 41

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  5. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  6. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  7. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  8. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  9. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  10. 2022Confident Adaptive Language Modelingcited by 297
  11. 2022Conformal Risk Controlcited by 290
  12. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 278
29 more
  1. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 256
  2. 2016Hyperparameter optimization with approximate gradientcited by 231
  3. 2021SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimizationcited by 229
  4. 2020Estimating means of bounded random variables by bettingcited by 164
  5. 2023Conformal Language Modelingcited by 163
  6. 2019Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentationcited by 162
  7. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  8. 2022Game-theoretic statistics and safe anytime-valid inferencecited by 145
  9. 2022Safe Planning in Dynamic Environments using Conformal Predictioncited by 112
  10. 2014Freeze-Thaw Bayesian Optimizationcited by 86
  11. 2023LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimizationcited by 86
  12. 2023Modular Deep Learningcited by 82
  13. 2021Fair Conformal Predictors for Applications in Medical Imagingcited by 74
  14. 2021There are natural scores: Full comment on Shafer, "Testing by betting: A strategy for statistical and scientific communication"cited by 49
  15. 2020Admissible anytime-valid sequential inference must rely on nonnegative martingalescited by 43
  16. 2021Nonparametric Two-Sample Testing by Bettingcited by 36
  17. 2023Provably safe systems: the only path to controllable AGIcited by 29
  18. 2022E-detectors: a nonparametric framework for sequential change detectioncited by 27
  19. 2024In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimizationcited by 26
  20. 2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scalingcited by 21
  21. 2023Forking Uncertainties: Reliable Prediction and Model Predictive Control with Sequence Models via Conformal Risk Controlcited by 19
  22. 2021Modular Meta-Learning for Power Control via Random Edge Graph Neural Networkscited by 13
  23. 2021A unified framework for bandit multiple testingcited by 8
  24. 2019Wireless Edge Computing with Latency and Reliability Guaranteescited by 6
  25. 2024Quantile Learn-Then-Test: Quantile-Based Risk Control for Hyperparameter Optimizationcited by 6
  26. 2020False (and Missed) Discoveries in Financial Economicscited by 3
  27. 2023When is Offline Policy Selection Sample Efficient for Reinforcement Learning?cited by 3
  28. 2024OREO: O-RAN intElligence Orchestration of xApp-based network servicescited by 3
  29. 2016Greedy-Knapsack Algorithm for Optimal Downlink Resource Allocation in LTE Networkscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 7.

13 papers.

Sort:

By area:

  1. 2025Decision Theoretic Foundations for Conformal Prediction: Optimal Uncertainty Quantification for Risk-Averse Agentscited by 37
  2. 2025Conformal Calibration: Ensuring the Reliability of Black-Box AI in Wireless Systemscited by 10
  3. 2025On Continuous Monitoring of Risk Violations under Unknown Shiftcited by 9
  4. 2026Optimal Decision-Making Based on Prediction Setscited by 4
  5. 2025Ensuring Reliability via Hyperparameter Selection: Review and Advancescited by 3
  6. 2025Reliable LLM-Based Edge-Cloud-Expert Cascades for Telecom Knowledge Systemscited by 2
  7. 2026Prediction-Powered Risk Monitoring of Deployed Models for Detecting Harmful Distribution Shiftscited by 2
  8. 2025Multi-Objective Hyperparameter Selection via Hypothesis Testing on Reliability Graphscited by 1
  9. 2026Conformal Risk-Averse Decision Making with Action Conditional Guaranteecited by 1
  10. 2026ProxyGuard: Direct Reliability Inference for Randomized Data Release Mechanisms with Shared Targetscited by 0
  11. 2026Should I Use This Synthetic Dataset for Training? How to Test with Minimal Real Datacited by 0
  12. 2026Certified AI Triage of ICU Alarmscited by 0
1 more
  1. 2026Evidence-Gated Research: Statistically Controlled Model Adoption in Adaptive Searchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning from the Test: Self-Referential Differential Testing for Deep RL Agents
  2. 2025Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL
  3. 2023Reinforcement Learning from Statistical Feedback: the Journey from AB Testing to ANT Testing
  4. 2026TTVS: Boosting Self-Exploring Reinforcement Learning via Test-time Variational Synthesis
  5. 2026Small RL Controller, Large Language Model: RL-Guided Adaptive Sampling for Test-Time Scaling
  6. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  7. 2024Reinforcement Learning for Online Testing of Autonomous Driving Systems: a Replication and Extension Study
  8. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL