mlsanityData to 14 Oct 2026

Multi-Objective Hyperparameter Selection via Hypothesis Testing on Reliability Graphs

arXiv 2501.13018 ↗First posted 22 Jan 2025cs.LG cs.IT math.ITcited by 1

Not on the map. Why?

Abstract (arXiv)

The selection of hyperparameters, such as prompt templates in large language models (LLMs), must often strike a balance between reliability and cost. In many cases, structural relationships between the expected reliability levels of the hyperparameters can be inferred from prior information and held-out data -- e.g., longer prompt templates may be more detailed and thus more reliable. However, existing hyperparameter selection methods either do not provide formal reliability guarantees or are unable to incorporate structured knowledge in the hyperparameter space. This paper introduces reliability graph-based Pareto testing (RG-PT), a novel multi-objective hyperparameter selection framework that maintains formal reliability guarantees in terms of false discovery rate (FDR), while accounting for known relationships among hyperparameters via a directed acyclic graph. Edges in the graph reflect expected reliability and cost trade-offs among hyperparameters, which are inferred via the Bradley-Terry (BT) ranking model from prior information and held-out data. Experimental evaluations demonstrate that RG-PT significantly outperforms existing methods such as learn-then-test (LTT) and Pareto testing (PT) through a more efficient exploration of the hyperparameter space.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 40 of its 90 references that are in the map. First posted on arXiv 22 Jan 2025.

Builds on

40 of 90 references.

References · 40

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2024A Survey on LLM-as-a-Judgecited by 1,423
  4. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  5. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  6. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  7. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 699
  8. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  9. 2019Conformalized Quantile Regressioncited by 629
  10. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 470
  11. 2017A Tutorial on Thompson Samplingcited by 433
  12. 2020Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlationcited by 425
28 more
  1. 2019Fair Resource Allocation in Federated Learningcited by 394
  2. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  3. 2023GPTScore: Evaluate as You Desirecited by 361
  4. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
  5. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 341
  6. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 271
  7. 2015Non-stochastic Best Arm Identification and Hyperparameter Optimizationcited by 221
  8. 2020On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practicecited by 182
  9. 2019Sparse Sequence-to-Sequence Modelscited by 170
  10. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  11. 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 145
  12. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 112
  13. 2018A Flexible Framework for Multi-Objective Bayesian Optimization using Random Scalarizationscited by 99
  14. 2015Predictive Entropy Search for Multi-objective Bayesian Optimizationcited by 88
  15. 2021Model Complexity of Deep Learning: A Surveycited by 65
  16. 2023SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformercited by 60
  17. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 59
  18. 2022Understanding O-RAN: Architecture, Interfaces, Algorithms, Security, and Research Challengescited by 55
  19. 2021Effective Sparsification of Neural Networks with Global Sparsity Constraintcited by 45
  20. 2022Efficiently Controlling Multiple Risks with Pareto Testingcited by 22
  21. 2024Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selectioncited by 15
  22. 2021Smoothing and Shrinking the Sparse Seq2Seq Search Spacecited by 11
  23. 2017A unified treatment of multiple testing with prior knowledge using the p-filtercited by 10
  24. 2015Bayesian Optimization of Text Representationscited by 6
  25. 2024Quantile Learn-Then-Test: Quantile-Based Risk Control for Hyperparameter Optimizationcited by 6
  26. 2016The Control of the False Discovery Rate in Fixed Sequence Multiple Testingcited by 2
  27. 2024Statistically Valid Information Bottleneck via Multiple Hypothesis Testingcited by 2
  28. 2018An LP-based hyperparameter optimization model for language modelingcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Ensuring Reliability via Hyperparameter Selection: Review and Advancescited by 3