Dealing with Sparse Rewards in Continuous Control Robotics via Heavy-Tailed Policies
Not on the map. Why?
Abstract (arXiv)
In this paper, we present a novel Heavy-Tailed Stochastic Policy Gradient (HT-PSG) algorithm to deal with the challenges of sparse rewards in continuous control problems. Sparse reward is common in continuous control robotics tasks such as manipulation and navigation, and makes the learning problem hard due to non-trivial estimation of value functions over the state space. This demands either reward shaping or expert demonstrations for the sparse reward environment. However, obtaining high-quality demonstrations is quite expensive and sometimes even impossible. We propose a heavy-tailed policy parametrization along with a modified momentum-based policy gradient tracking scheme (HT-SPG) to induce a stable exploratory behavior to the algorithm. The proposed algorithm does not require access to expert demonstrations. We test the performance of HT-SPG on various benchmark tasks of continuous control with sparse rewards such as 1D Mario, Pathological Mountain Car, Sparse Pendulum in OpenAI Gym, and Sparse MuJoCo environments (Hopper-v2). We show consistent performance improvement across all tasks in terms of high average cumulative reward. HT-SPG also demonstrates improved convergence speed with minimum samples, thereby emphasizing the sample efficiency of our proposed algorithm.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 23 of its 43 references that are in the map. First posted on arXiv 12 Jun 2022.
Builds on
23 of 43 references.
References · 23
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2016OpenAI Gymcited by 1,684
- 2016Generative Adversarial Imitation Learningcited by 1,576
- 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
- 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
- 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
- 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
- 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
- 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
- 2017Parameter Space Noise for Explorationcited by 158
- 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 117
- 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
11 more
- 2018Stochastic Variance-Reduced Policy Gradientcited by 103
- 2018Motion Planning Among Dynamic, Decision-Making Agents with Deep Reinforcement Learningcited by 101
- 2019Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewardscited by 58
- 2022Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstrationcited by 54
- 2020Stochastic Recursive Momentum for Policy Gradient Methodscited by 22
- 2019The problem with DDPG: understanding failures in deterministic environments with sparse rewardscited by 20
- 2020Guided Exploration with Proximal Policy Optimization using a Single Demonstrationcited by 19
- 2021On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Controlcited by 12
- 2019Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrationscited by 10
- 2020On Reward Shaping for Mobile Robot Navigation: A Reinforcement Learning and SLAM Based Approachcited by 5
- 2020Policy Gradient from Demonstration and Curiositycited by 4
Built on by
Citing papers per year: 2023: 2.
2 papers.