# Learning to Explore with Meta-Policy Gradient

arXiv: https://arxiv.org/abs/1803.05044 · HTML: https://mlsanity.com/paper/1803.05044/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Mar 2018.

## Built on by (most cited first, 9 of 9)

- [A Survey of Exploration Methods in Reinforcement Learning](https://mlsanity.com/paper/2109.00157/) (2021)
- [Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments](https://mlsanity.com/paper/2101.08152/) (2021)
- [Improving Context-Based Meta-Reinforcement Learning with Self-Supervised Trajectory Contrastive Learning](https://mlsanity.com/paper/2103.06386/) (2021)
- [Learning an Explicit Hyperparameter Prediction Function Conditioned on Tasks](https://mlsanity.com/paper/2107.02378/) (2021)
- [Autonomous Driving at Unsignalized Intersections: A Review of Decision-Making Challenges and Reinforcement Learning-Based Solutions](https://mlsanity.com/paper/2409.13144/) (2024)
- [Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.12957/) (2021)
- [Optimistic Meta-Gradients](https://mlsanity.com/paper/2301.03236/) (2023)
- [Meta-DiffuB: A Contextualized Sequence-to-Sequence Text Diffusion Model with Meta-Exploration](https://mlsanity.com/paper/2410.13201/) (2024)
- [MESA: Cooperative Meta-Exploration in Multi-Agent Learning through Exploiting State-Action Space Structure](https://mlsanity.com/paper/2405.00902/) (2024)

## Abstract (arXiv)

The performance of off-policy learning, including deep Q-learning and deep deterministic policy gradient (DDPG), critically depends on the choice of the exploration policy. Existing exploration methods are mostly based on adding noise to the on-going actor policy and can only explore local regions close to what the actor policy dictates. In this work, we develop a simple meta-policy gradient algorithm that allows us to adaptively learn the exploration policy in DDPG. Our algorithm allows us to train flexible exploration behaviors that are independent of the actor policy, yielding a global exploration that significantly speeds up the learning process. With an extensive study, we show that our method significantly improves the sample-efficiency of DDPG on a variety of reinforcement learning tasks.