mlsanityData to 14 Oct 2026

Stepsize Learning for Policy Gradient Methods in Contextual Markov Decision Processes

arXiv 2306.07741 ↗First posted 13 Jun 2023cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Policy-based algorithms are among the most widely adopted techniques in model-free RL, thanks to their strong theoretical groundings and good properties in continuous action spaces. Unfortunately, these methods require precise and problem-specific hyperparameter tuning to achieve good performance, and tend to struggle when asked to accomplish a series of heterogeneous tasks. In particular, the selection of the step size has a crucial impact on their ability to learn a highly performing policy, affecting the speed and the stability of the training process, and often being the main culprit for poor results. In this paper, we tackle these issues with a Meta Reinforcement Learning approach, by introducing a new formulation, known as meta-MDP, that can be used to solve any hyperparameter selection problem in RL with contextual processes. After providing a theoretical Lipschitz bound to the difference of performance in different tasks, we adopt the proposed framework to train a batch RL algorithm to dynamically recommend the most adequate step size for different policies and tasks. In conclusion, we present an experimental campaign to show the advantages of selecting an adaptive learning rate in heterogeneous environments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 64 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

30 of 64 references.

References · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2016OpenAI Gymcited by 1,736
  6. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 699
  9. 2016Learning to learn by gradient descent by gradient descentcited by 620
  10. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  11. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  12. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
18 more
  1. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
  2. 2015Contextual Markov Decision Processescited by 220
  3. 2016Learning to Optimizecited by 195
  4. 2018Meta-Gradient Reinforcement Learningcited by 159
  5. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  6. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 114
  7. 2019Meta-Curvaturecited by 70
  8. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 70
  9. 2022Automated Dynamic Algorithm Configurationcited by 39
  10. 2019Hyp-RL : Hyperparameter Optimization by Reinforcement Learningcited by 36
  11. 2019Deep Reinforcement Learning using Genetic Algorithm for Parameter Optimizationcited by 24
  12. 2020Importance of Tuning Hyperparameters of Machine Learning Algorithmscited by 20
  13. 2019A Meta-MDP Approach to Exploration for Lifelong Reinforcement Learningcited by 18
  14. 2018Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methodscited by 17
  15. 2017Reinforcement Learning for Learning Rate Controlcited by 16
  16. 2019Fast Efficient Hyperparameter Tuning for Policy Gradientscited by 6
  17. 2017Online Learning of a Memory for Learning Ratescited by 4
  18. 2021Online hyperparameter optimization by real-time recurrent learningcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator
  2. 2024Theoretical Analysis of Meta Reinforcement Learning: Generalization Bounds and Convergence Guarantees
  3. 2022Discovered Policy Optimisation
  4. 2024MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning
  5. 2026Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Training
  6. 2025Scalable Multi-Objective and Meta Reinforcement Learning via Gradient Estimation
  7. 2022Model-Based Offline Meta-Reinforcement Learning with Regularization
  8. 2024Meta-Gradient Search Control: A Method for Improving the Efficiency of Dyna-style Planning