# Bayesian Optimization for Iterative Learning

arXiv: https://arxiv.org/abs/1909.09593 · HTML: https://mlsanity.com/paper/1909.09593/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Sep 2019.

## Built on by (most cited first, 19 of 19)

- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety](https://mlsanity.com/paper/2104.14235/) (2021)
- [PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning](https://mlsanity.com/paper/2306.12370/) (2023)
- [NAS-Bench-x11 and the Power of Learning Curves](https://mlsanity.com/paper/2111.03602/) (2021)
- [Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL](https://mlsanity.com/paper/2106.15883/) (2021)
- [Non-smooth Bayesian Optimization in Tuning Problems](https://mlsanity.com/paper/2109.07563/) (2021)
- [Bayesian Optimization Over Iterative Learners with Structured Responses: A Budget-aware Planning Approach](https://mlsanity.com/paper/2206.12708/) (2022)
- [High-dimensional Bayesian Optimization via Covariance Matrix Adaptation Strategy](https://mlsanity.com/paper/2402.03104/) (2024)
- [Efficient Hyperparameter Optimization for Physics-based Character Animation](https://mlsanity.com/paper/2104.12365/) (2021)
- [Using AntiPatterns to avoid MLOps Mistakes](https://mlsanity.com/paper/2107.00079/) (2021)
- [Amortized Auto-Tuning: Cost-Efficient Bayesian Transfer Optimization for Hyperparameter Recommendation](https://mlsanity.com/paper/2106.09179/) (2021)
- [Domain Knowledge Injection in Bayesian Search for New Materials](https://mlsanity.com/paper/2311.15162/) (2023)
- [Adaptive Q-Network: On-the-fly Target Selection for Deep Reinforcement Learning](https://mlsanity.com/paper/2405.16195/) (2024)
- [Cost-Efficient Online Hyperparameter Optimization](https://mlsanity.com/paper/2101.06590/) (2021)
- [Bayesian inference for data-efficient, explainable, and safe robotic motion planning: A review](https://mlsanity.com/paper/2307.08024/) (2023)
- [A Trajectory-Based Bayesian Approach to Multi-Objective Hyperparameter Optimization with Epoch-Aware Trade-Offs](https://mlsanity.com/paper/2405.15303/) (2024)
- [Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Review](https://mlsanity.com/paper/2505.07911/) (2025)
- [Genealogical Population-Based Training for Hyperparameter Optimization](https://mlsanity.com/paper/2109.14925/) (2021)
- [Process-constrained batch Bayesian approaches for yield optimization in multi-reactor systems](https://mlsanity.com/paper/2408.02551/) (2024)

## Abstract (arXiv)

The performance of deep (reinforcement) learning systems crucially depends on the choice of hyperparameters. Their tuning is notoriously expensive, typically requiring an iterative training process to run for numerous steps to convergence. Traditional tuning algorithms only consider the final performance of hyperparameters acquired after many expensive iterations and ignore intermediate information from earlier training steps. In this paper, we present a Bayesian optimization (BO) approach which exploits the iterative structure of learning algorithms for efficient hyperparameter tuning. We propose to learn an evaluation function compressing learning progress at any stage of the training process into a single numeric score according to both training success and stability. Our BO framework is then balancing the benefit of assessing a hyperparameter setting over additional training steps against their computation cost. We further increase model efficiency by selectively including scores from different training steps for any evaluated hyperparameter set. We demonstrate the efficiency of our algorithm by tuning hyperparameters for the training of deep reinforcement learning agents and convolutional neural networks. Our algorithm outperforms all existing baselines in identifying optimal hyperparameters in minimal time.