# Recurrent Hypernetworks are Surprisingly Strong in Meta-RL

arXiv: https://arxiv.org/abs/2309.14970 · HTML: https://mlsanity.com/paper/2309.14970/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 31 references that are in the map.
- First posted on arXiv 26 Sep 2023.

## Builds on (most cited first, 24 of 24)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Meta-Learning with Latent Embedding Optimization](https://mlsanity.com/paper/1807.05960/) (2018)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Meta Networks](https://mlsanity.com/paper/1703.00837/) (2017)
- [VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning](https://mlsanity.com/paper/1910.08348/) (2019)
- [Bayesian Model-Agnostic Meta-Learning](https://mlsanity.com/paper/1806.03836/) (2018)
- [Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs](https://mlsanity.com/paper/2110.05038/) (2021)
- [A Tutorial on Meta-Reinforcement Learning](https://mlsanity.com/paper/2301.08028/) (2023)
- [Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation](https://mlsanity.com/paper/1910.13616/) (2019)
- [Meta reinforcement learning as task inference](https://mlsanity.com/paper/1905.06424/) (2019)
- [Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices](https://mlsanity.com/paper/2008.02790/) (2020)
- [Hypernetworks in Meta-Reinforcement Learning](https://mlsanity.com/paper/2210.11348/) (2022)
- [Generalization of Reinforcement Learners with Working and Episodic Memory](https://mlsanity.com/paper/1910.13406/) (2019)
- [Recomposing the Reinforcement Learning Building Blocks with Hypernetworks](https://mlsanity.com/paper/2106.06842/) (2021)
- [Rapid Task-Solving in Novel Environments](https://mlsanity.com/paper/2006.03662/) (2020)
- [HyperDynamics: Meta-Learning Object and Agent Dynamics with Hypernetworks](https://mlsanity.com/paper/2103.09439/) (2021)
- [HyperMAML: Few-Shot Adaptation of Deep Models with Hypernetworks](https://mlsanity.com/paper/2205.15745/) (2022)
- [Learning Adaptive Exploration Strategies in Dynamic Environments Through Informed Policy Regularization](https://mlsanity.com/paper/2005.02934/) (2020)
- [Linear Representation Meta-Reinforcement Learning for Instant Adaptation](https://mlsanity.com/paper/2101.04750/) (2021)

## Built on by (most cited first, 16 of 16)

- [XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning](https://mlsanity.com/paper/2406.08973/) (2024)
- [Distilling Morphology-Conditioned Hypernetworks for Efficient Universal Morphology Control](https://mlsanity.com/paper/2402.06570/) (2024)
- [HyperMARL: Adaptive Hypernetworks for Multi-Agent RL](https://mlsanity.com/paper/2412.04233/) (2024)
- [HyperVLA: Efficient Inference in Vision-Language-Action Models via Hypernetworks](https://mlsanity.com/paper/2510.04898/) (2025)
- [DynaMITE-RL: A Dynamic Model for Improved Temporal Meta-Reinforcement Learning](https://mlsanity.com/paper/2402.15957/) (2024)
- [Metalic: Meta-Learning In-Context with Protein Language Models](https://mlsanity.com/paper/2410.08355/) (2024)
- [Capability-Aware Shared Hypernetworks for Flexible Heterogeneous Multi-Robot Coordination](https://mlsanity.com/paper/2501.06058/) (2025)
- [SplAgger: Split Aggregation for Meta-Reinforcement Learning](https://mlsanity.com/paper/2403.03020/) (2024)
- [Enhancing Diversity in Bayesian Deep Learning via Hyperspherical Energy Minimization of CKA](https://mlsanity.com/paper/2411.00259/) (2024)
- [Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learning](https://mlsanity.com/paper/2412.14834/) (2024)
- [TIMRL: A Novel Meta-Reinforcement Learning Framework for Non-Stationary and Multi-Task Environments](https://mlsanity.com/paper/2501.07146/) (2025)
- [Hypernetworks for Model-Heterogeneous Personalized Federated Learning](https://mlsanity.com/paper/2507.22330/) (2025)
- [Hypernetworks for Dynamic Feature Selection](https://mlsanity.com/paper/2605.12278/) (2026)
- [MATE: Solving Contextual Markov Decision Processes with Memory of Accumulated Transition Embeddings](https://mlsanity.com/paper/2605.17431/) (2026)
- [Knowledge Reutilization in Meta-Reinforcement Learning](https://mlsanity.com/paper/2606.18132/) (2026)
- [MOCHA: Multi-Objective Co-Design using Hypernetwork Architectures](https://mlsanity.com/paper/2609.30570/) (2026)

## Abstract (arXiv)

Deep reinforcement learning (RL) is notoriously impractical to deploy due to sample inefficiency. Meta-RL directly addresses this sample inefficiency by learning to perform few-shot learning when a distribution of related tasks is available for meta-training. While many specialized meta-RL methods have been proposed, recent work suggests that end-to-end learning in conjunction with an off-the-shelf sequential model, such as a recurrent network, is a surprisingly strong baseline. However, such claims have been controversial due to limited supporting evidence, particularly in the face of prior work establishing precisely the opposite. In this paper, we conduct an empirical investigation. While we likewise find that a recurrent network can achieve strong performance, we demonstrate that the use of hypernetworks is crucial to maximizing their potential. Surprisingly, when combined with hypernetworks, the recurrent baselines that are far simpler than existing specialized methods actually achieve the strongest performance of all methods evaluated. We provide code at https://github.com/jacooba/hyper.