# Hyp-RL : Hyperparameter Optimization by Reinforcement Learning

arXiv: https://arxiv.org/abs/1906.11527 · HTML: https://mlsanity.com/paper/1906.11527/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jun 2019.

## Built on by (most cited first, 30 of 36)

- [HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenML](https://mlsanity.com/paper/2106.06257/) (2021)
- [Revisiting Weighted Aggregation in Federated Learning with Neural Networks](https://mlsanity.com/paper/2302.10911/) (2023)
- [Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety](https://mlsanity.com/paper/2104.14235/) (2021)
- [B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasoners](https://mlsanity.com/paper/2412.17256/) (2024)
- [A survey on multi-objective hyperparameter optimization algorithms for Machine Learning](https://mlsanity.com/paper/2111.13755/) (2021)
- [SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Models](https://mlsanity.com/paper/2411.02433/) (2024)
- [FaiR-IoT: Fairness-aware Human-in-the-Loop Reinforcement Learning for Harnessing Human Variability in Personalized IoT](https://mlsanity.com/paper/2103.16033/) (2021)
- [Reinforcement Learning Meets Visual Odometry](https://mlsanity.com/paper/2407.15626/) (2024)
- [Automated Deep Learning: Neural Architecture Search Is Not the End](https://mlsanity.com/paper/2112.09245/) (2021)
- [Automated Reinforcement Learning: An Overview](https://mlsanity.com/paper/2201.05000/) (2022)
- [Hyperparameter Tuning for Deep Reinforcement Learning Applications](https://mlsanity.com/paper/2201.11182/) (2022)
- [Quant 4.0: Engineering Quantitative Investment with Automated, Explainable and Knowledge-driven Artificial Intelligence](https://mlsanity.com/paper/2301.04020/) (2022)
- [ArchGym: An Open-Source Gymnasium for Machine Learning Assisted Architecture Design](https://mlsanity.com/paper/2306.08888/) (2023)
- [Deep Ranking Ensembles for Hyperparameter Optimization](https://mlsanity.com/paper/2303.15212/) (2023)
- [A Game of Bundle Adjustment -- Learning Efficient Convergence](https://mlsanity.com/paper/2308.13270/) (2023)
- [Hyperparameter Optimization with Differentiable Metafeatures](https://mlsanity.com/paper/2102.03776/) (2021)
- [BFTrainer: Low-Cost Training of Neural Networks on Unfillable Supercomputer Nodes](https://mlsanity.com/paper/2106.12091/) (2021)
- [AoI-minimizing Scheduling in UAV-relayed IoT Networks](https://mlsanity.com/paper/2107.05181/) (2021)
- [Online Meta-Learning for Model Update Aggregation in Federated Learning for Click-Through Rate Prediction](https://mlsanity.com/paper/2209.00629/) (2022)
- [Reinforcement Teaching](https://mlsanity.com/paper/2204.11897/) (2022)
- [Magpie: Automatically Tuning Static Parameters for Distributed File Systems using Deep Reinforcement Learning](https://mlsanity.com/paper/2207.09298/) (2022)
- [ULTHO: Ultra-Lightweight yet Efficient Hyperparameter Optimization in Deep Reinforcement Learning](https://mlsanity.com/paper/2503.06101/) (2025)
- [Improving Hyperparameter Optimization by Planning Ahead](https://mlsanity.com/paper/2110.08028/) (2021)
- [Hyper-parameter optimization based on soft actor critic and hierarchical mixture regularization](https://mlsanity.com/paper/2112.04084/) (2021)
- [Discrete Simulation Optimization for Tuning Machine Learning Method Hyperparameters](https://mlsanity.com/paper/2201.05978/) (2022)
- [A Practical AoI Scheduler in IoT Networks with Relays](https://mlsanity.com/paper/2203.04227/) (2022)
- [A Framework for History-Aware Hyperparameter Optimisation in Reinforcement Learning](https://mlsanity.com/paper/2303.05186/) (2023)
- [Stepsize Learning for Policy Gradient Methods in Contextual Markov Decision Processes](https://mlsanity.com/paper/2306.07741/) (2023)
- [Scope Loss for Imbalanced Classification and RL Exploration](https://mlsanity.com/paper/2308.04024/) (2023)
- [HyperQ-Opt: Q-learning for Hyperparameter Optimization](https://mlsanity.com/paper/2412.17765/) (2024)

## Abstract (arXiv)

Hyperparameter tuning is an omnipresent problem in machine learning as it is an integral aspect of obtaining the state-of-the-art performance for any model. Most often, hyperparameters are optimized just by training a model on a grid of possible hyperparameter values and taking the one that performs best on a validation sample (grid search). More recently, methods have been introduced that build a so-called surrogate model that predicts the validation loss for a specific hyperparameter setting, model and dataset and then sequentially select the next hyperparameter to test, based on a heuristic function of the expected value and the uncertainty of the surrogate model called acquisition function (sequential model-based Bayesian optimization, SMBO). In this paper we model the hyperparameter optimization problem as a sequential decision problem, which hyperparameter to test next, and address it with reinforcement learning. This way our model does not have to rely on a heuristic acquisition function like SMBO, but can learn which hyperparameters to test next based on the subsequent reduction in validation loss they will eventually lead to, either because they yield good models themselves or because they allow the hyperparameter selection policy to build a better surrogate model that is able to choose better hyperparameters later on. Experiments on a large battery of 50 data sets demonstrate that our method outperforms the state-of-the-art approaches for hyperparameter learning.