# The problem with DDPG: understanding failures in deterministic environments with sparse rewards

arXiv: https://arxiv.org/abs/1911.11679 · HTML: https://mlsanity.com/paper/1911.11679/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Nov 2019.

## Built on by (most cited first, 20 of 20)

- [Continuous Doubly Constrained Batch Reinforcement Learning](https://mlsanity.com/paper/2102.09225/) (2021)
- [On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Control](https://mlsanity.com/paper/2106.08414/) (2021)
- [CACTO: Continuous Actor-Critic with Trajectory Optimization -- Towards global optimality](https://mlsanity.com/paper/2211.06625/) (2022)
- [On the Hidden Biases of Policy Mirror Ascent in Continuous Action Spaces](https://mlsanity.com/paper/2201.12332/) (2022)
- [Policy Optimization in Dynamic Bayesian Network Hybrid Models of Biomanufacturing Processes](https://mlsanity.com/paper/2105.06543/) (2021)
- [Deep Reinforcement Learning-based UAV Navigation and Control: A Soft Actor-Critic with Hindsight Experience Replay Approach](https://mlsanity.com/paper/2106.01016/) (2021)
- [SMPL: Simulated Industrial Manufacturing and Process Control Learning Environments](https://mlsanity.com/paper/2206.08851/) (2022)
- [How to Choose a Reinforcement-Learning Algorithm](https://mlsanity.com/paper/2407.20917/) (2024)
- [GUIDE: Real-Time Human-Shaped Agents](https://mlsanity.com/paper/2410.15181/) (2024)
- [Divide & Conquer Imitation Learning](https://mlsanity.com/paper/2204.07404/) (2022)
- [Dealing with Sparse Rewards in Continuous Control Robotics via Heavy-Tailed Policies](https://mlsanity.com/paper/2206.05652/) (2022)
- [ETGL-DDPG: A Deep Deterministic Policy Gradient Algorithm for Sparse Reward Continuous Control](https://mlsanity.com/paper/2410.05225/) (2024)
- [Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Methods](https://mlsanity.com/paper/2601.22970/) (2026)
- [Selection-Expansion: A Unifying Framework for Motion-Planning and Diversity Search Algorithms](https://mlsanity.com/paper/2104.04768/) (2021)
- [Using Deep Reinforcement Learning for Zero Defect Smart Forging](https://mlsanity.com/paper/2201.10268/) (2022)
- [Understanding Reinforcement Learning Algorithms: The Progress from Basic Q-learning to Proximal Policy Optimization](https://mlsanity.com/paper/2304.00026/) (2023)
- [Robust Deep Reinforcement Learning Scheduling via Weight Anchoring](https://mlsanity.com/paper/2304.10176/) (2023)
- [What deep reinforcement learning tells us about human motor learning and vice-versa](https://mlsanity.com/paper/2208.10892/) (2022)
- [DynaMark: A Reinforcement Learning Framework for Dynamic Watermarking in Industrial Machine Tool Controllers](https://mlsanity.com/paper/2508.21797/) (2025)
- [Dual Advantage Fields](https://mlsanity.com/paper/2606.04188/) (2026)

## Abstract (arXiv)

In environments with continuous state and action spaces, state-of-the-art actor-critic reinforcement learning algorithms can solve very complex problems, yet can also fail in environments that seem trivial, but the reason for such failures is still poorly understood. In this paper, we contribute a formal explanation of these failures in the particular case of sparse reward and deterministic environments. First, using a very elementary control problem, we illustrate that the learning process can get stuck into a fixed point corresponding to a poor solution. Then, generalizing from the studied example, we provide a detailed analysis of the underlying mechanisms which results in a new understanding of one of the convergence regimes of these algorithms. The resulting perspective casts a new light on already existing solutions to the issues we have highlighted, and suggests other potential approaches.