# Guided Meta-Policy Search

arXiv: https://arxiv.org/abs/1904.00956 · HTML: https://mlsanity.com/paper/1904.00956/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Apr 2019.

## Built on by (most cited first, 26 of 26)

- [MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale](https://mlsanity.com/paper/2104.08212/) (2021)
- [Dropout Q-Functions for Doubly Efficient Reinforcement Learning](https://mlsanity.com/paper/2110.02034/) (2021)
- [Offline Meta-Reinforcement Learning with Advantage Weighting](https://mlsanity.com/paper/2008.06043/) (2020)
- [AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning](https://mlsanity.com/paper/2107.02729/) (2021)
- [How Fine-Tuning Allows for Effective Meta-Learning](https://mlsanity.com/paper/2105.02221/) (2021)
- [Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents](https://mlsanity.com/paper/2102.02926/) (2021)
- [When Is Generalizable Reinforcement Learning Tractable?](https://mlsanity.com/paper/2101.00300/) (2021)
- [CoMPS: Continual Meta Policy Search](https://mlsanity.com/paper/2112.04467/) (2021)
- [Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusion](https://mlsanity.com/paper/2407.10973/) (2024)
- [MetaTrader: An Reinforcement Learning Approach Integrating Diverse Policies for Portfolio Optimization](https://mlsanity.com/paper/2210.01774/) (2022)
- [IMM: An Imitative Reinforcement Learning Approach with Predictive Representation Learning for Automatic Market Making](https://mlsanity.com/paper/2308.08918/) (2023)
- [Dream to Adapt: Meta Reinforcement Learning by Latent Context Imagination and MDP Imagination](https://mlsanity.com/paper/2311.06673/) (2023)
- [A Meta-Reinforcement Learning Approach to Process Control](https://mlsanity.com/paper/2103.14060/) (2021)
- [Continual Robot Learning using Self-Supervised Task Inference](https://mlsanity.com/paper/2309.04974/) (2023)
- [Improved Robustness and Safety for Pre-Adaptation of Meta Reinforcement Learning with Prior Regularization](https://mlsanity.com/paper/2108.08448/) (2021)
- [Learning Policy Committees for Effective Personalization in MDPs with Diverse Tasks](https://mlsanity.com/paper/2503.01885/) (2025)
- [Continual Meta-Reinforcement Learning for UAV-Aided Vehicular Wireless Networks](https://mlsanity.com/paper/2207.06131/) (2022)
- [Sequential Decision Making with Expert Demonstrations under Unobserved Heterogeneity](https://mlsanity.com/paper/2404.07266/) (2024)
- [Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator](https://mlsanity.com/paper/2410.09728/) (2024)
- [Off-Policy Meta-Reinforcement Learning Based on Feature Embedding Spaces](https://mlsanity.com/paper/2101.01883/) (2021)
- [Enhanced Meta Reinforcement Learning using Demonstrations in Sparse Reward Environments](https://mlsanity.com/paper/2209.13048/) (2022)
- [Efficient Sparse-Reward Goal-Conditioned Reinforcement Learning with a High Replay Ratio and Regularization](https://mlsanity.com/paper/2312.05787/) (2023)
- [Model Based Meta Learning of Critics for Policy Gradients](https://mlsanity.com/paper/2204.02210/) (2022)
- [Implicit Training of Energy Model for Structure Prediction](https://mlsanity.com/paper/2211.11649/) (2022)
- [Smart Sampling: Self-Attention and Bootstrapping for Improved Ensembled Q-Learning](https://mlsanity.com/paper/2405.08252/) (2024)
- [Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data](https://mlsanity.com/paper/2501.07346/) (2025)

## Abstract (arXiv)

Reinforcement learning (RL) algorithms have demonstrated promising results on complex tasks, yet often require impractical numbers of samples since they learn from scratch. Meta-RL aims to address this challenge by leveraging experience from previous tasks so as to more quickly solve new tasks. However, in practice, these algorithms generally also require large amounts of on-policy experience during the meta-training process, making them impractical for use in many problems. To this end, we propose to learn a reinforcement learning procedure in a federated way, where individual off-policy learners can solve the individual meta-training tasks, and then consolidate these solutions into a single meta-learner. Since the central meta-learner learns by imitating the solutions to the individual tasks, it can accommodate either the standard meta-RL problem setting or a hybrid setting where some or all tasks are provided with example demonstrations. The former results in an approach that can leverage policies learned for previous tasks without significant amounts of on-policy data during meta-training, whereas the latter is particularly useful in cases where demonstrations are easy for a person to provide. Across a number of continuous control meta-RL problems, we demonstrate significant improvements in meta-RL sample efficiency in comparison to prior work as well as the ability to scale to domains with visual observations.