# What is Going on Inside Recurrent Meta Reinforcement Learning Agents?

arXiv: https://arxiv.org/abs/2104.14644 · HTML: https://mlsanity.com/paper/2104.14644/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 15 references that are in the map.
- First posted on arXiv 29 Apr 2021.

## Builds on (most cited first, 9 of 9)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Meta-Gradient Reinforcement Learning](https://mlsanity.com/paper/1805.09801/) (2018)
- [Evolved Policy Gradients](https://mlsanity.com/paper/1802.04821/) (2018)
- [Meta reinforcement learning as task inference](https://mlsanity.com/paper/1905.06424/) (2019)
- [Meta-learning of Sequential Strategies](https://mlsanity.com/paper/1905.03030/) (2019)
- [A Brief Look at Generalization in Visual Meta-Reinforcement Learning](https://mlsanity.com/paper/2006.07262/) (2020)

## Built on by (most cited first, 3 of 3)

- [A model-based approach to meta-Reinforcement Learning: Transformers and tree search](https://mlsanity.com/paper/2208.11535/) (2022)
- [Are LSTMs Good Few-Shot Learners?](https://mlsanity.com/paper/2310.14139/) (2023)
- [Meta-Reinforcement Learning with Discrete World Models for Adaptive Load Balancing](https://mlsanity.com/paper/2503.08872/) (2025)

## Abstract (arXiv)

Recurrent meta reinforcement learning (meta-RL) agents are agents that employ a recurrent neural network (RNN) for the purpose of "learning a learning algorithm". After being trained on a pre-specified task distribution, the learned weights of the agent's RNN are said to implement an efficient learning algorithm through their activity dynamics, which allows the agent to quickly solve new tasks sampled from the same distribution. However, due to the black-box nature of these agents, the way in which they work is not yet fully understood. In this study, we shed light on the internal working mechanisms of these agents by reformulating the meta-RL problem using the Partially Observable Markov Decision Process (POMDP) framework. We hypothesize that the learned activity dynamics is acting as belief states for such agents. Several illustrative experiments suggest that this hypothesis is true, and that recurrent meta-RL agents can be viewed as agents that learn to act optimally in partially observable environments consisting of multiple related tasks. This view helps in understanding their failure cases and some interesting model-based results reported in the literature.