# Guided Exploration with Proximal Policy Optimization using a Single Demonstration

arXiv: https://arxiv.org/abs/2007.03328 · HTML: https://mlsanity.com/paper/2007.03328/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jul 2020.

## Built on by (most cited first, 19 of 19)

- [Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstration](https://mlsanity.com/paper/2202.04628/) (2022)
- [Prioritized Experience-based Reinforcement Learning with Human Guidance for Autonomous Driving](https://mlsanity.com/paper/2109.12516/) (2021)
- [Refined Policy Distillation: From VLA Generalists to RL Experts](https://mlsanity.com/paper/2503.05833/) (2025)
- [ACEGEN: Reinforcement learning of generative chemical agents for drug discovery](https://mlsanity.com/paper/2405.04657/) (2024)
- [Proximal Policy Optimization with Adaptive Threshold for Symmetric Relative Density Ratio](https://mlsanity.com/paper/2203.09809/) (2022)
- [Combining imitation and deep reinforcement learning to accomplish human-level performance on a virtual foraging task](https://mlsanity.com/paper/2203.06250/) (2022)
- [Reinforcement Learning with Physics-Informed Symbolic Program Priors for Zero-Shot Wireless Indoor Navigation](https://mlsanity.com/paper/2506.22365/) (2025)
- [Learning Agentic Policy from Action Guidance](https://mlsanity.com/paper/2605.12004/) (2026)
- [Sample Dropout: A Simple yet Effective Variance Reduction Technique in Deep Policy Optimization](https://mlsanity.com/paper/2302.02299/) (2023)
- [REINFORCE-ING Chemical Language Models for Drug Discovery](https://mlsanity.com/paper/2501.15971/) (2025)
- [TADPO: Reinforcement Learning Goes Off-road](https://mlsanity.com/paper/2603.05995/) (2026)
- [Dealing with Sparse Rewards in Continuous Control Robotics via Heavy-Tailed Policies](https://mlsanity.com/paper/2206.05652/) (2022)
- [Trajectory-Oriented Policy Optimization with Sparse Rewards](https://mlsanity.com/paper/2401.02225/) (2024)
- [Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agents](https://mlsanity.com/paper/2604.13733/) (2026)
- [Policy Optimization with Smooth Guidance Learned from State-Only Demonstrations](https://mlsanity.com/paper/2401.00162/) (2023)
- [Preference-Guided Reinforcement Learning for Efficient Exploration](https://mlsanity.com/paper/2407.06503/) (2024)
- [TEACH: Temporal Variance-Driven Curriculum for Reinforcement Learning](https://mlsanity.com/paper/2512.22824/) (2025)
- [Match or Replay: Self Imitating Proximal Policy Optimization](https://mlsanity.com/paper/2603.27515/) (2026)
- [Toward an Energy-Optimized Operation of Data Centers Located in Wind Farms Using Reinforcement Learning](https://mlsanity.com/paper/2606.30316/) (2026)

## Abstract (arXiv)

Solving sparse reward tasks through exploration is one of the major challenges in deep reinforcement learning, especially in three-dimensional, partially-observable environments. Critically, the algorithm proposed in this article uses a single human demonstration to solve hard-exploration problems. We train an agent on a combination of demonstrations and own experience to solve problems with variable initial conditions. We adapt this idea and integrate it with the proximal policy optimization (PPO). The agent is able to increase its performance and to tackle harder problems by replaying its own past trajectories prioritizing them based on the obtained reward and the maximum value of the trajectory. We compare different variations of this algorithm to behavioral cloning on a set of hard-exploration tasks in the Animal-AI Olympics environment. To the best of our knowledge, learning a task in a three-dimensional environment with comparable difficulty has never been considered before using only one human demonstration.