Enhanced Meta Reinforcement Learning using Demonstrations in Sparse Reward Environments
Not on the map. Why?
Abstract (arXiv)
Meta reinforcement learning (Meta-RL) is an approach wherein the experience gained from solving a variety of tasks is distilled into a meta-policy. The meta-policy, when adapted over only a small (or just a single) number of steps, is able to perform near-optimally on a new, related task. However, a major challenge to adopting this approach to solve real-world problems is that they are often associated with sparse reward functions that only indicate whether a task is completed partially or fully. We consider the situation where some data, possibly generated by a sub-optimal agent, is available for each task. We then develop a class of algorithms entitled Enhanced Meta-RL using Demonstrations (EMRLD) that exploit this information even if sub-optimal to obtain guidance during training. We show how EMRLD jointly utilizes RL and supervised learning over the offline data to generate a meta-policy that demonstrates monotone performance improvements. We also develop a warm started variant called EMRLD-WS that is particularly efficient for sub-optimal demonstration data. Finally, we show that our EMRLD algorithms significantly outperform existing approaches in a variety of sparse reward environments, including that of a mobile robot.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 26 of its 50 references that are in the map. First posted on arXiv 26 Sep 2022.
Builds on
26 of 50 references.
References · 26
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2018On First-Order Meta-Learning Algorithmscited by 1,003
- 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
- 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
- 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
- 2016Learning to reinforcement learncited by 369
- 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
- 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
- 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
14 more
- 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
- 2018Meta-Reinforcement Learning of Structured Exploration Strategiescited by 128
- 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 106
- 2018ProMP: Proximal Meta-Policy Searchcited by 91
- 2020learn2learn: A Library for Meta-Learning Researchcited by 69
- 2022Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstrationcited by 54
- 2018DiCE: The Infinitely Differentiable Monte-Carlo Estimatorcited by 46
- 2018Some Considerations on Learning to Explore via Meta-Reinforcement Learningcited by 43
- 2021Offline Meta-Reinforcement Learning for Industrial Insertioncited by 37
- 2017Deep Q-learning from Demonstrationscited by 27
- 2019Guided Meta-Policy Searchcited by 26
- 2019Watch, Try, Learn: Meta-Learning from Demonstrations and Rewardcited by 26
- 2022All You Need Is Supervised Learning: From Imitation Learning to Meta-RL With Upside Down RLcited by 9
- 2021Hindsight Task Relabelling: Experience Replay for Sparse Reward Meta-RLcited by 8
Built on by
Citing papers per year: 2026: 1.
1 paper.