# Policy Gradient from Demonstration and Curiosity

arXiv: https://arxiv.org/abs/2004.10430 · HTML: https://mlsanity.com/paper/2004.10430/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Apr 2020.

## Built on by (most cited first, 4 of 4)

- [Beyond Joint Demonstrations: Personalized Expert Guidance for Efficient Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2403.08936/) (2024)
- [DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learning](https://mlsanity.com/paper/2309.08925/) (2023)
- [Guiding Reinforcement Learning Using Uncertainty-Aware Large Language Models](https://mlsanity.com/paper/2411.14457/) (2024)
- [Dealing with Sparse Rewards in Continuous Control Robotics via Heavy-Tailed Policies](https://mlsanity.com/paper/2206.05652/) (2022)

## Abstract (arXiv)

With reinforcement learning, an agent could learn complex behaviors from high-level abstractions of the task. However, exploration and reward shaping remained challenging for existing methods, especially in scenarios where the extrinsic feedback was sparse. Expert demonstrations have been investigated to solve these difficulties, but a tremendous number of high-quality demonstrations were usually required. In this work, an integrated policy gradient algorithm was proposed to boost exploration and facilitate intrinsic reward learning from only limited number of demonstrations. We achieved this by reformulating the original reward function with two additional terms, where the first term measured the Jensen-Shannon divergence between current policy and the expert, and the second term estimated the agent's uncertainty about the environment. The presented algorithm was evaluated on a range of simulated tasks with sparse extrinsic reward signals where only one single demonstrated trajectory was provided to each task, superior exploration efficiency and high average return were demonstrated in all tasks. Furthermore, it was found that the agent could imitate the expert's behavior and meanwhile sustain high return.