mlsanityData to 14 Oct 2026

No More Pesky Hyperparameters: Offline Hyperparameter Tuning for RL

arXiv 2205.08716 ↗First posted 18 May 2022cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

The performance of reinforcement learning (RL) agents is sensitive to the choice of hyperparameters. In real-world settings like robotics or industrial control systems, however, testing different hyperparameter configurations directly on the environment can be financially prohibitive, dangerous, or time consuming. We propose a new approach to tune hyperparameters from offline logs of data, to fully specify the hyperparameters for an RL agent that learns online in the real world. The approach is conceptually simple: we first learn a model of the environment from the offline data, which we call a calibration model, and then simulate learning in the calibration model to identify promising hyperparameters. We identify several criteria to make this strategy effective, and develop an approach that satisfies these criteria. We empirically investigate the method in a variety of settings to identify when it is effective and when it fails.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 44 of its 79 references that are in the map. First posted on arXiv 18 May 2022.

Builds on

44 of 79 references.

References · 44

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Deep Reinforcement Learning that Matterscited by 754
  3. 2018A Tutorial on Bayesian Optimizationcited by 744
  4. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 699
  5. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  6. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  7. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 470
  8. 2017Population Based Training of Neural Networkscited by 367
  9. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 228
  10. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  11. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 190
  12. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
32 more
  1. 2018Meta-Gradient Reinforcement Learningcited by 159
  2. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 149
  3. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 136
  4. 2016Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasetscited by 132
  5. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 114
  6. 2020Learning Stable Deep Dynamics Modelscited by 111
  7. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 91
  8. 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 86
  9. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  10. 2017Learning human behaviors from motion capture by adversarial imitationcited by 77
  11. 2018Benchmarking Reinforcement Learning Algorithms on Real-World Robotscited by 65
  12. 2020Provably Efficient Online Hyperparameter Optimization with Population-Based Banditscited by 63
  13. 2016Self-Correcting Models for Model-Based Reinforcement Learningcited by 43
  14. 2016Unifying task specification in reinforcement learningcited by 41
  15. 2020Optimizing for the Future in Non-Stationary MDPscited by 37
  16. 2021Domain Adaptation In Reinforcement Learning Via Latent Unified State Representationcited by 35
  17. 2020Offline Policy Selection under Uncertaintycited by 31
  18. 2020Selective Dyna-style Planning Under Limited Model Capacitycited by 30
  19. 2020Towards Safe Policy Improvement for Non-Stationary MDPscited by 26
  20. 2018Learning from Demonstration in the Wildcited by 25
  21. 2020Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimizationcited by 22
  22. 2019Smoothing Policies and Safe Policy Gradientscited by 21
  23. 2016A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learningcited by 20
  24. 2018Organizing Experience: A Deeper Look at Replay Mechanisms for Sample-based Planning in Continuous State Domainscited by 19
  25. 2020Learning Dynamics Models with Stable Invariant Setscited by 18
  26. 2019Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Futurecited by 17
  27. 2020Forethought and Hindsight in Credit Assignmentcited by 16
  28. 2021Almost Surely Stable Deep Dynamicscited by 16
  29. 2020Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategiescited by 11
  30. 2019Meta-descent for Online, Continual Predictioncited by 10
  31. 2020Learning Constrained Adaptive Differentiable Predictive Control Policies With Guaranteescited by 10
  32. 2016Adaptive Lambda Least-Squares Temporal Difference Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 0; 2025: 1; 2026: 2.

6 papers.

  1. 2023Bootstrapped Training of Score-Conditioned Generator for Offline Design of Biological Sequencescited by 23
  2. 2023Proper Laplacian Representation Learningcited by 11
  3. 2025A Clean Slate for Offline Reinforcement Learningcited by 11
  4. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directionscited by 5
  5. 2023End-to-end Offline Reinforcement Learning for Glycemia Controlcited by 2
  6. 2026A Diffusion-Model Subpopulation Digital Twin for Mobile Health Deployment: A Case Study on the HeartSteps Interventioncited by 1