# On the Practical Consistency of Meta-Reinforcement Learning Algorithms

arXiv: https://arxiv.org/abs/2112.00478 · HTML: https://mlsanity.com/paper/2112.00478/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 1 Dec 2021.

## Builds on (most cited first, 18 of 18)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design](https://mlsanity.com/paper/2012.02096/) (2020)
- [VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning](https://mlsanity.com/paper/1910.08348/) (2019)
- [Meta-Gradient Reinforcement Learning](https://mlsanity.com/paper/1805.09801/) (2018)
- [Offline Meta-Reinforcement Learning with Advantage Weighting](https://mlsanity.com/paper/2008.06043/) (2020)
- [ProMP: Proximal Meta-Policy Search](https://mlsanity.com/paper/1810.06784/) (2018)
- [Evolved Policy Gradients](https://mlsanity.com/paper/1802.04821/) (2018)
- [Discovering Reinforcement Learning Algorithms](https://mlsanity.com/paper/2007.08794/) (2020)
- [Improving Generalization in Meta Reinforcement Learning using Learned Objectives](https://mlsanity.com/paper/1910.04098/) (2019)
- [Meta-Q-Learning](https://mlsanity.com/paper/1910.00125/) (2019)
- [Some Considerations on Learning to Explore via Meta-Reinforcement Learning](https://mlsanity.com/paper/1803.01118/) (2018)
- [Unsupervised Meta-Learning for Reinforcement Learning](https://mlsanity.com/paper/1806.04640/) (2018)
- [Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling](https://mlsanity.com/paper/2006.07178/) (2020)

## Built on by (most cited first, 6 of 6)

- [Model-Free Opponent Shaping](https://mlsanity.com/paper/2205.01447/) (2022)
- [On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learning](https://mlsanity.com/paper/2206.03271/) (2022)
- [Metalic: Meta-Learning In-Context with Protein Language Models](https://mlsanity.com/paper/2410.08355/) (2024)
- [Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator](https://mlsanity.com/paper/2410.09728/) (2024)
- [Bayesian Meta-Reinforcement Learning with Laplace Variational Recurrent Networks](https://mlsanity.com/paper/2505.18591/) (2025)
- [RL3: Boosting Meta Reinforcement Learning via RL inside RL2](https://mlsanity.com/paper/2306.15909/) (2023)

## Abstract (arXiv)

Consistency is the theoretical property of a meta learning algorithm that ensures that, under certain assumptions, it can adapt to any task at test time. An open question is whether and how theoretical consistency translates into practice, in comparison to inconsistent algorithms. In this paper, we empirically investigate this question on a set of representative meta-RL algorithms. We find that theoretically consistent algorithms can indeed usually adapt to out-of-distribution (OOD) tasks, while inconsistent ones cannot, although they can still fail in practice for reasons like poor exploration. We further find that theoretically inconsistent algorithms can be made consistent by continuing to update all agent components on the OOD tasks, and adapt as well or better than originally consistent ones. We conclude that theoretical consistency is indeed a desirable property, and inconsistent meta-RL algorithms can easily be made consistent to enjoy the same benefits.