mlsanityData to 7 Oct 2026

Automatic tuning of hyper-parameters of reinforcement learning algorithms using Bayesian optimization with behavioral cloning

arXiv 2112.08094 ↗First posted 15 Dec 2021cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Optimal setting of several hyper-parameters in machine learning algorithms is key to make the most of available data. To this aim, several methods such as evolutionary strategies, random search, Bayesian optimization and heuristic rules of thumb have been proposed. In reinforcement learning (RL), the information content of data gathered by the learning agent while interacting with its environment is heavily dependent on the setting of many hyper-parameters. Therefore, the user of an RL algorithm has to rely on search-based optimization methods, such as grid search or the Nelder-Mead simplex algorithm, that are very inefficient for most RL tasks, slows down significantly the learning curve and leaves to the user the burden of purposefully biasing data gathering. In this work, in order to make an RL algorithm more user-independent, a novel approach for autonomous hyper-parameter setting using Bayesian optimization is proposed. Data from past episodes and different hyper-parameter values are used at a meta-learning level by performing behavioral cloning which helps improving the effectiveness in maximizing a reinforcement learning variant of an acquisition function. Also, by tightly integrating Bayesian optimization in a reinforcement learning agent design, the number of state transitions needed to converge to the optimal policy for a given task is reduced. Computational experiments reveal promising results compared to other manual tweaking and optimization-based approaches which highlights the benefits of changing the algorithm hyper-parameters to increase the information content of generated data.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 74 references that are in the map. First posted on arXiv 15 Dec 2021.

Builds on

37 of 74 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul

Other references · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  8. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  9. 2015Prioritized Experience Replaycited by 1,070
  10. 2017Deep Reinforcement Learning that Matterscited by 727
  11. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  12. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
24 more
  1. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  2. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
  3. 2017Population Based Training of Neural Networkscited by 351
  4. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 301
  5. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  6. 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 277
  7. 2015Non-stochastic Best Arm Identification and Hyperparameter Optimizationcited by 219
  8. 2018Meta-Gradient Reinforcement Learningcited by 156
  9. 2017Deep Exploration via Randomized Value Functionscited by 155
  10. 2019Behaviour Suite for Reinforcement Learningcited by 129
  11. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 92
  12. 2018Bayesian Optimization of Combinatorial Structurescited by 64
  13. 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
  14. 2018Bayesian Optimization in AlphaGocited by 55
  15. 2016CMA-ES for Hyperparameter Optimization of Deep Neural Networkscited by 47
  16. 2019Discovery of Useful Questions as Auxiliary Taskscited by 45
  17. 2019Off-Policy Actor-Critic with Shared Experience Replaycited by 39
  18. 2020Sample-Efficient Automated Deep Reinforcement Learningcited by 31
  19. 2015How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategiescited by 26
  20. 2016A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learningcited by 20
  21. 2020Online Meta-Critic Learning for Off-Policy Actor-Critic Methodscited by 17
  22. 2020Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategiescited by 9
  23. 2019Beyond Exponentially Discounted Sum: Automatic Learning of Return Functioncited by 7
  24. 2018Towards Autonomous Reinforcement Learning: Automatic Setting of Hyper-parameters using Bayesian Optimizationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Hyperparameter Optimization for Driving Strategies Based on Reinforcement Learning
  2. 2023A Strong Baseline for Batch Imitation Learning
  3. 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review
  4. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data
  5. 2023Stepsize Learning for Policy Gradient Methods in Contextual Markov Decision Processes
  6. 2026Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief
  7. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
  8. 2025A Clean Slate for Offline Reinforcement Learning