# Dynamic Learning Rate for Deep Reinforcement Learning: A Bandit Approach

arXiv: https://arxiv.org/abs/2410.12598 · HTML: https://mlsanity.com/paper/2410.12598/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 27 of its 56 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 27 of 27)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [Non-stochastic Best Arm Identification and Hyperparameter Optimization](https://mlsanity.com/paper/1502.07943/) (2015)
- [Dopamine: A Research Framework for Deep Reinforcement Learning](https://mlsanity.com/paper/1812.06110/) (2018)
- [A Definition of Continual Reinforcement Learning](https://mlsanity.com/paper/2307.11046/) (2023)
- [Playing FPS Games with Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05521/) (2016)
- [Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progress](https://mlsanity.com/paper/2206.01626/) (2022)
- [Provably Efficient Online Hyperparameter Optimization with Population-Based Bandits](https://mlsanity.com/paper/2002.02518/) (2020)
- [TorchRL: A data-driven decision-making library for PyTorch](https://mlsanity.com/paper/2306.00577/) (2023)
- [Normalization and effective learning rates in reinforcement learning](https://mlsanity.com/paper/2407.01800/) (2024)
- [Tactical Optimism and Pessimism for Deep Reinforcement Learning](https://mlsanity.com/paper/2102.03765/) (2021)
- [Meta-Gradient Reinforcement Learning with an Objective Discovered Online](https://mlsanity.com/paper/2007.08433/) (2020)
- [Time-Varying Gaussian Process Bandit Optimization](https://mlsanity.com/paper/1601.06650/) (2016)
- [The Difficulty of Passive Learning in Deep Reinforcement Learning](https://mlsanity.com/paper/2110.14020/) (2021)
- [Adam with Bandit Sampling for Deep Learning](https://mlsanity.com/paper/2010.12986/) (2020)
- [Efficient and Generalizable Tuning Strategies for Stochastic Gradient MCMC](https://mlsanity.com/paper/2105.13059/) (2021)
- [Step-size Optimization for Continual Learning](https://mlsanity.com/paper/2401.17401/) (2024)
- [Learning with Random Learning Rates](https://mlsanity.com/paper/1810.01322/) (2018)
- [Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?](https://mlsanity.com/paper/2408.07588/) (2024)

## Abstract (arXiv)

In deep Reinforcement Learning (RL), the learning rate critically influences both stability and performance, yet its optimal value shifts during training as the environment and policy evolve. Standard decay schedulers assume monotonic convergence and often misalign with these dynamics, leading to premature or delayed adjustments. We introduce LRRL, a meta-learning approach that dynamically selects the learning rate based on policy performance rather than training steps. LRRL adaptively favors rates that improve returns, remaining robust even when the candidate set includes values that individually cause divergence. Across Atari and MuJoCo benchmarks, LRRL achieves performance competitive with or superior to tuned baselines and standard schedulers. Our findings position LRRL as a practical solution for adapting to non-stationary objectives in deep RL.