Efficient Sparse-Reward Goal-Conditioned Reinforcement Learning with a High Replay Ratio and Regularization
Not on the map. Why?
Abstract (arXiv)
Reinforcement learning (RL) methods with a high replay ratio (RR) and regularization have gained interest due to their superior sample efficiency. However, these methods have mainly been developed for dense-reward tasks. In this paper, we aim to extend these RL methods to sparse-reward goal-conditioned tasks. We use Randomized Ensemble Double Q-learning (REDQ) (Chen et al., 2021), an RL method with a high RR and regularization. To apply REDQ to sparse-reward goal-conditioned tasks, we make the following modifications to it: (i) using hindsight experience replay and (ii) bounding target Q-values. We evaluate REDQ with these modifications on 12 sparse-reward goal-conditioned tasks of Robotics (Plappert et al., 2018), and show that it achieves about 2 × better sample efficiency than previous state-of-the-art (SoTA) RL methods. Furthermore, we reconsider the necessity of specific components of REDQ and simplify it by removing unnecessary ones. The simplified REDQ with our modifications achieves ~ 8 × better sample efficiency than the SoTA methods in 4 Fetch tasks of Robotics.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 55 of its 67 references that are in the map. First posted on arXiv 10 Dec 2023.
Builds on
55 of 67 references.
References · 55
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2016Layer Normalizationcited by 5,579
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
- 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
- 2017Hindsight Experience Replaycited by 1,096
- 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
- 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
- 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
38 more
- 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
- 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
- 2023Language to Rewards for Robotic Skill Synthesiscited by 307
- 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
- 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
- 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
- 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
- 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
- 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 207
- 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
- 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
- 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
- 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
- 2018Self-Imitation Learningcited by 135
- 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
- 2023Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learningcited by 111
- 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
- 2021Reward (Mis)design for Autonomous Drivingcited by 102
- 2023A Tutorial on Meta-Reinforcement Learningcited by 102
- 2016Model-Free Episodic Controlcited by 100
- 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 100
- 2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 76
- 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 76
- 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
- 2019Exploration via Hindsight Goal Generationcited by 53
- 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
- 2018Episodic Memory Deep Q-Networkscited by 41
- 2019Maximum Entropy-Regularized Multi-Goal Reinforcement Learningcited by 39
- 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
- 2023Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learningcited by 29
- 2019Curiosity-Driven Experience Prioritization via Density Estimationcited by 27
- 2019Guided Meta-Policy Searchcited by 26
- 2023Learning and Adapting Agile Locomotion Skills by Transferring Experiencecited by 25
- 2016Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tighteningcited by 21
- 2023Grow Your Limits: Continuous Improvement with Real-World RL for Robotic Locomotioncited by 21
- 2023Accelerating Exploration with Unlabeled Prior Datacited by 21
- 2018Energy-Based Hindsight Experience Prioritizationcited by 20
- 2021Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learningcited by 15
50 most cited of 55 All 55 (JSON)
Built on by
Citing papers per year: 2024: 1.
1 paper.