mlsanityData to 13 Oct 2026

ETGL-DDPG: A Deep Deterministic Policy Gradient Algorithm for Sparse Reward Continuous Control

arXiv 2410.05225 ↗First posted 7 Oct 2024cs.LG cs.RO stat.MLcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We consider deep deterministic policy gradient (DDPG) in the context of reinforcement learning with sparse rewards. To enhance exploration, we introduce a search procedure, emphεt-greedy, which generates exploratory options for exploring less-visited states. We prove that search using ε t-greedy has polynomial sample complexity under mild MDP assumptions. To more efficiently use the information provided by rewarded transitions, we develop a new dual experience replay buffer framework, GDRB, and implement longest n-step returns. The resulting algorithm, ETGL-DDPG, integrates all three techniques: bmε t-greedy, GDRB, and Longest n-step, into DDPG. We evaluate ETGL-DDPG on standard benchmarks and demonstrate that it outperforms DDPG, as well as other state-of-the-art methods, across all tested sparse-reward continuous environments. Ablation studies further highlight how each strategy individually enhances the performance of DDPG in this setting.

Key facts. Cited by 2 papers in the map (data through 13 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 44 of its 65 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

44 of 65 references.

References · 44

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,656
  2. 2015Continuous control with deep reinforcement learningcited by 3,290
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,221
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,310
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,223
  6. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,317
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,180
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,175
  9. 2017Hindsight Experience Replaycited by 1,104
  10. 2015Prioritized Experience Replaycited by 1,078
  11. 2018Exploration by Random Network Distillationcited by 971
  12. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 777
32 more
  1. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 699
  2. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  3. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  4. 2020First return, then explorecited by 257
  5. 2017Count-Based Exploration with Neural Density Modelscited by 255
  6. 2020Never Give Up: Learning Directed Exploration Strategiescited by 211
  7. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  8. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 192
  9. 2017Parameter Space Noise for Explorationcited by 158
  10. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 118
  11. 2020Learning to Utilize Shaping Rewards: A New Approach of Reward Shapingcited by 113
  12. 2017A Deeper Look at Experience Replaycited by 104
  13. 2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 76
  14. 2023Generative Flow Networks as Entropy-Regularized RLcited by 61
  15. 2018GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithmscited by 40
  16. 2021Self-Supervised Online Reward Shaping in Sparse-Reward Environmentscited by 34
  17. 2020Softmax Deep Double Deterministic Policy Gradientscited by 31
  18. 2023Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learningcited by 28
  19. 2021Decoupled Reinforcement Learning to Stabilise Intrinsically-Motivated Explorationcited by 25
  20. 2017ES Is More Than Just a Traditional Finite-Difference Approximatorcited by 24
  21. 2019Competitive Experience Replaycited by 23
  22. 2019The problem with DDPG: understanding failures in deterministic environments with sparse rewardscited by 20
  23. 2020Understanding and Mitigating the Limitations of Prioritized Experience Replaycited by 16
  24. 2020The Effect of Multi-step Methods on Overestimation in Deep Reinforcement Learningcited by 14
  25. 2022Monte Carlo Augmented Actor-Critic for Sparse Reward Deep Reinforcement Learning from Suboptimal Demonstrationscited by 14
  26. 2018When Simple Exploration is Sample Efficient: Identifying Sufficient Conditions for Random Exploration to Yield PAC RL Algorithmscited by 11
  27. 2020Learning of Long-Horizon Sparse-Reward Robotic Manipulator Tasks with Base Controllerscited by 7
  28. 2019Improved Exploration through Latent Trajectory Optimization in Deep Deterministic Policy Gradientcited by 4
  29. 2022Handling Sparse Rewards in Reinforcement Learning Using Model Predictive Controlcited by 4
  30. 2022Event Tables for Efficient Experience Replaycited by 4
  31. 2024Informativeness of Reward Functions in Reinforcement Learningcited by 3
  32. 2013L'evy flights in human behavior and cognitioncited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learningcited by 0
  2. 2026Chess on Ice: Curling Tactical Decision-Making via Backward Induction and Deep Reinforcement Learningcited by 0