# Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methods

arXiv: https://arxiv.org/abs/1810.02525 · HTML: https://mlsanity.com/paper/1810.02525/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2018.

## Built on by (most cited first, 17 of 17)

- [Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective](https://mlsanity.com/paper/2105.05246/) (2021)
- [On Proximal Policy Optimization's Heavy-tailed Gradients](https://mlsanity.com/paper/2102.10264/) (2021)
- [Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlo](https://mlsanity.com/paper/2305.18246/) (2023)
- [Aligning Language Models with Offline Learning from Human Feedback](https://mlsanity.com/paper/2308.12050/) (2023)
- [Can Learned Optimization Make Reinforcement Learning Less Difficult?](https://mlsanity.com/paper/2407.07082/) (2024)
- [Learning to Optimize for Reinforcement Learning](https://mlsanity.com/paper/2302.01470/) (2023)
- [Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timesteps](https://mlsanity.com/paper/2412.17113/) (2024)
- [Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers](https://mlsanity.com/paper/2505.20128/) (2025)
- [Efficiently Quantifying Individual Agent Importance in Cooperative MARL](https://mlsanity.com/paper/2312.08466/) (2023)
- [Design and Planning of Flexible Mobile Micro-Grids Using Deep Reinforcement Learning](https://mlsanity.com/paper/2212.04136/) (2022)
- [Parameter-free Gradient Temporal Difference Learning](https://mlsanity.com/paper/2105.04129/) (2021)
- [Application of Monte Carlo Stochastic Optimization (MOST) to Deep Learning](https://mlsanity.com/paper/2109.02441/) (2021)
- [Task-optimal data-driven surrogate models for eNMPC via differentiable simulation and optimization](https://mlsanity.com/paper/2403.14425/) (2024)
- [A Scalable Finite Difference Method for Deep Reinforcement Learning](https://mlsanity.com/paper/2210.07487/) (2022)
- [Stepsize Learning for Policy Gradient Methods in Contextual Markov Decision Processes](https://mlsanity.com/paper/2306.07741/) (2023)
- [Time-Scale Separation in Q-Learning: Extending TD(triangle) for Action-Value Function Decomposition](https://mlsanity.com/paper/2411.14019/) (2024)
- [Segmenting Action-Value Functions Over Time-Scales in SARSA via TD(Delta)](https://mlsanity.com/paper/2411.14783/) (2024)

## Abstract (arXiv)

Recent analyses of certain gradient descent optimization methods have shown that performance can degrade in some settings - such as with stochasticity or implicit momentum. In deep reinforcement learning (Deep RL), such optimization methods are often used for training neural networks via the temporal difference error or policy gradient. As an agent improves over time, the optimization target changes and thus the loss landscape (and local optima) change. Due to the failure modes of those methods, the ideal choice of optimizer for Deep RL remains unclear. As such, we provide an empirical analysis of the effects that a wide range of gradient descent optimizers and their hyperparameters have on policy gradient methods, a subset of Deep RL algorithms, for benchmark continuous control tasks. We find that adaptive optimizers have a narrow window of effective learning rates, diverging in other cases, and that the effectiveness of momentum varies depending on the properties of the environment. Our analysis suggests that there is significant interplay between the dynamics of the environment and Deep RL algorithm properties which aren't necessarily accounted for by traditional adaptive gradient methods. We provide suggestions for optimal settings of current methods and further lines of research based on our findings.