mlsanityData to 7 Oct 2026

On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning

arXiv 2102.13651 ↗First posted 26 Feb 2021cs.LG cs.AI cs.NEcited by 59

Not on the map. Why?

Abstract (arXiv)

Model-based Reinforcement Learning (MBRL) is a promising framework for learning control in a data-efficient manner. MBRL algorithms can be fairly complex due to the separate dynamics modeling and the subsequent planning algorithm, and as a result, they often possess tens of hyperparameters and architectural choices. For this reason, MBRL typically requires significant human expertise before it can be applied to new problems and domains. To alleviate this problem, we propose to use automatic hyperparameter optimization (HPO). We demonstrate that this problem can be tackled effectively with automated HPO, which we demonstrate to yield significantly improved performance compared to human experts. In addition, we show that tuning of several MBRL hyperparameters dynamically, i.e. during the training itself, further improves the performance compared to using static hyperparameters which are kept fixed for the whole training. Finally, our experiments provide valuable insights into the effects of several hyperparameters, such as plan horizon or learning rate and their influence on the stability of training and resulting rewards.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 17 of its 28 references that are in the map. First posted on arXiv 26 Feb 2021.

Builds on

17 of 28 references.

References · 17

  1. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  2. 2017Deep Reinforcement Learning that Matterscited by 727
  3. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  4. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
  5. 2017Population Based Training of Neural Networkscited by 351
  6. 2015Non-stochastic Best Arm Identification and Hyperparameter Optimizationcited by 219
  7. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  8. 2018Differentiable MPC for End-to-end Planning and Controlcited by 178
  9. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  10. 2016Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasetscited by 131
  11. 2017On the State of the Art of Evaluation in Neural Language Modelscited by 103
  12. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 92
5 more
  1. 2020Objective Mismatch in Model-based Reinforcement Learningcited by 87
  2. 2018Learning Navigation Behaviors End-to-End with AutoRLcited by 70
  3. 2016Multi-fidelity Gaussian Process Bandit Optimisationcited by 31
  4. 2018Learning to Design RNAcited by 27
  5. 2019Ctrl-Z: Recovering from Instability in Reinforcement Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 17; 2023: 14; 2024: 8; 2025: 6; 2026: 2.

59 papers.

Sort:

By area:

  1. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  2. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  3. 2022Defining and Characterizing Reward Hackingcited by 184
  4. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 96
  5. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  6. 2022A Survey on Model-based Reinforcement Learningcited by 65
  7. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 58
  8. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learningcited by 50
  9. 2022Investigating Compounding Prediction Errors in Learned Dynamics Modelscited by 47
  10. 2021MBRL-Lib: A Modular Library for Model-based Reinforcement Learningcited by 41
  11. 2022Contextualize Me -- The Case for Context in Reinforcement Learningcited by 41
  12. 2023PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learningcited by 31
38 more
  1. 2022Bayesian Generational Population-Based Trainingcited by 20
  2. 2022Policy Optimization with Linear Temporal Logic Constraintscited by 19
  3. 2025Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classificationcited by 19
  4. 2023Eventual Discounting Temporal Logic Counterfactual Experience Replaycited by 18
  5. 2023Ensemble Reinforcement Learning: A Surveycited by 16
  6. 2023Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoMLcited by 16
  7. 2024Dreaming of Many Worlds: Learning Contextual World Models Aids Zero-Shot Generalizationcited by 16
  8. 2021On Effective Scheduling of Model-based Reinforcement Learningcited by 13
  9. 2021Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?cited by 11
  10. 2022User-Interactive Offline Reinforcement Learningcited by 11
  11. 2023Adaptive Ensemble Q-learning: Minimizing Estimation Bias via Error Feedbackcited by 9
  12. 2021Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRLcited by 8
  13. 2021On-Policy Model Errors in Reinforcement Learningcited by 6
  14. 2021Adaptively Calibrated Critic Estimates for Deep Reinforcement Learningcited by 5
  15. 2021Episodic Policy Gradient Trainingcited by 5
  16. 2022A Simple Decentralized Cross-Entropy Methodcited by 5
  17. 2023Evolving Populations of Diverse RL Agents with MAP-Elitescited by 5
  18. 2021Differentiable Architecture Search for Reinforcement Learningcited by 4
  19. 2023Dynamic Update-to-Data Ratio: Minimizing World Model Overfittingcited by 4
  20. 2023Multi-Objective Population Based Trainingcited by 4
  21. 2023Robustness of Algorithms for Causal Structure Learning to Hyperparameter Choicecited by 4
  22. 2024ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learningcited by 3
  23. 2025Bootstrapped Reward Shapingcited by 3
  24. 2022ARLO: A Framework for Automated Reinforcement Learningcited by 2
  25. 2023A systematic study comparing hyperparameter optimization engines on tabular datacited by 2
  26. 2024Fast Benchmarking of Asynchronous Multi-Fidelity Optimization on Zero-Cost Benchmarkscited by 2
  27. 2024Decomposing Control Lyapunov Functions for Efficient Reinforcement Learningcited by 2
  28. 2022Practitioner Motives to Use Different Hyperparameter Optimization Methodscited by 1
  29. 2022Evolving Pareto-Optimal Actor-Critic Algorithms for Generalizability and Stabilitycited by 1
  30. 2022T3VIP: Transformation-based 3D Video Predictioncited by 1
  31. 2023Learning Environment for the Air Domain (LEAD)cited by 1
  32. 2024Deep autoregressive density nets vs neural ensembles for model-based offline reinforcement learningcited by 1
  33. 2024Efficient Off-Policy Learning for High-Dimensional Action Spacescited by 1
  34. 2025ULTHO: Ultra-Lightweight yet Efficient Hyperparameter Optimization in Deep Reinforcement Learningcited by 1
  35. 2025Hyperparameter Optimisation with Practical Interpretability and Explanation Methods in Probabilistic Curriculum Learningcited by 1
  36. 2025Multiple-Frequencies Population-Based Trainingcited by 1
  37. 2021Automatic tuning of hyper-parameters of reinforcement learning algorithms using Bayesian optimization with behavioral cloningcited by 0
  38. 2022Survival Prediction of Children Undergoing Hematopoietic Stem Cell Transplantation Using Different Machine Learning Classifiers by Performing Chi-squared Test and Hyper-parameter Optimization: A Retrospective Analysiscited by 0

50 most cited of 59 All 59 (JSON)