mlsanityData to 7 Oct 2026

On the Practical Consistency of Meta-Reinforcement Learning Algorithms

arXiv 2112.00478 ↗First posted 1 Dec 2021cs.LG cs.AI stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

Consistency is the theoretical property of a meta learning algorithm that ensures that, under certain assumptions, it can adapt to any task at test time. An open question is whether and how theoretical consistency translates into practice, in comparison to inconsistent algorithms. In this paper, we empirically investigate this question on a set of representative meta-RL algorithms. We find that theoretically consistent algorithms can indeed usually adapt to out-of-distribution (OOD) tasks, while inconsistent ones cannot, although they can still fail in practice for reasons like poor exploration. We further find that theoretically inconsistent algorithms can be made consistent by continuing to update all agent components on the OOD tasks, and adapt as well or better than originally consistent ones. We conclude that theoretical consistency is indeed a desirable property, and inconsistent meta-RL algorithms can easily be made consistent to enjoy the same benefits.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 18 of its 29 references that are in the map. First posted on arXiv 1 Dec 2021.

Builds on

18 of 29 references.

References · 18

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  4. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  5. 2016Learning to reinforcement learncited by 369
  6. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  7. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  8. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  9. 2018Meta-Gradient Reinforcement Learningcited by 156
  10. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 92
  11. 2018ProMP: Proximal Meta-Policy Searchcited by 91
  12. 2018Evolved Policy Gradientscited by 86
6 more
  1. 2020Discovering Reinforcement Learning Algorithmscited by 75
  2. 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
  3. 2019Meta-Q-Learningcited by 62
  4. 2018Some Considerations on Learning to Explore via Meta-Reinforcement Learningcited by 43
  5. 2018Unsupervised Meta-Learning for Reinforcement Learningcited by 38
  6. 2020Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabelingcited by 24

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1; 2024: 2; 2025: 1.

6 papers.

  1. 2022Model-Free Opponent Shapingcited by 43
  2. 2022On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learningcited by 18
  3. 2024Metalic: Meta-Learning In-Context with Protein Language Modelscited by 3
  4. 2024Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparatorcited by 2
  5. 2025Bayesian Meta-Reinforcement Learning with Laplace Variational Recurrent Networkscited by 1
  6. 2023RL3: Boosting Meta Reinforcement Learning via RL inside RL2cited by 0