# Abstract Demonstrations and Adaptive Exploration for Efficient and Stable Multi-step Sparse Reward Reinforcement Learning

arXiv: https://arxiv.org/abs/2207.09243 · HTML: https://mlsanity.com/paper/2207.09243/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 25 references that are in the map.
- First posted on arXiv 19 Jul 2022.

## Builds on (most cited first, 10 of 10)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Modular Multitask Reinforcement Learning with Policy Sketches](https://mlsanity.com/paper/1611.01796/) (2016)
- [Automatic Curriculum Learning For Deep RL: A Short Survey](https://mlsanity.com/paper/2003.04664/) (2020)
- [TACO: Learning Task Decomposition via Temporal Alignment for Control](https://mlsanity.com/paper/1803.01840/) (2018)
- [DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrations](https://mlsanity.com/paper/1710.05421/) (2017)
- [Sub-Goal Trees -- a Framework for Goal-Based Reinforcement Learning](https://mlsanity.com/paper/2002.12361/) (2020)
- [An Open-Source Multi-Goal Reinforcement Learning Environment for Robotic Manipulation with Pybullet](https://mlsanity.com/paper/2105.05985/) (2021)

## Built on by (most cited first, 1 of 1)

- [Relay Hindsight Experience Replay: Self-Guided Continual Reinforcement Learning for Sequential Object Manipulation Tasks with Sparse Rewards](https://mlsanity.com/paper/2208.00843/) (2022)

## Abstract (arXiv)

Although Deep Reinforcement Learning (DRL) has been popular in many disciplines including robotics, state-of-the-art DRL algorithms still struggle to learn long-horizon, multi-step and sparse reward tasks, such as stacking several blocks given only a task-completion reward signal. To improve learning efficiency for such tasks, this paper proposes a DRL exploration technique, termed A^2, which integrates two components inspired by human experiences: Abstract demonstrations and Adaptive exploration. A^2 starts by decomposing a complex task into subtasks, and then provides the correct orders of subtasks to learn. During training, the agent explores the environment adaptively, acting more deterministically for well-mastered subtasks and more stochastically for ill-learnt subtasks. Ablation and comparative experiments are conducted on several grid-world tasks and three robotic manipulation tasks. We demonstrate that A^2 can aid popular DRL algorithms (DQN, DDPG, and SAC) to learn more efficiently and stably in these environments.