# Hypernetworks in Meta-Reinforcement Learning

arXiv: https://arxiv.org/abs/2210.11348 · HTML: https://mlsanity.com/paper/2210.11348/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 20 Oct 2022.

## Builds on (most cited first, 23 of 23)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Meta-Learning with Latent Embedding Optimization](https://mlsanity.com/paper/1807.05960/) (2018)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Meta-SGD: Learning to Learn Quickly for Few-Shot Learning](https://mlsanity.com/paper/1707.09835/) (2017)
- [Meta Networks](https://mlsanity.com/paper/1703.00837/) (2017)
- [Distral: Robust Multitask Reinforcement Learning](https://mlsanity.com/paper/1707.04175/) (2017)
- [Learning to Optimize](https://mlsanity.com/paper/1606.01885/) (2016)
- [Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning](https://mlsanity.com/paper/1511.06342/) (2015)
- [Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation](https://mlsanity.com/paper/1910.13616/) (2019)
- [Differentiable plasticity: training plastic neural networks with backpropagation](https://mlsanity.com/paper/1804.02464/) (2018)
- [Meta reinforcement learning as task inference](https://mlsanity.com/paper/1905.06424/) (2019)
- [Meta-Learning through Hebbian Plasticity in Random Networks](https://mlsanity.com/paper/2007.02686/) (2020)
- [Recomposing the Reinforcement Learning Building Blocks with Hypernetworks](https://mlsanity.com/paper/2106.06842/) (2021)
- [HyperDynamics: Meta-Learning Object and Agent Dynamics with Hypernetworks](https://mlsanity.com/paper/2103.09439/) (2021)
- [HyperMAML: Few-Shot Adaptation of Deep Models with Hypernetworks](https://mlsanity.com/paper/2205.15745/) (2022)
- [Linear Representation Meta-Reinforcement Learning for Instant Adaptation](https://mlsanity.com/paper/2101.04750/) (2021)

## Built on by (most cited first, 30 of 42)

- [A Brief Review of Hypernetworks in Deep Learning](https://mlsanity.com/paper/2306.06955/) (2023)
- [SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass](https://mlsanity.com/paper/2602.06358/) (2026)
- [Universal Morphology Control via Contextual Modulation](https://mlsanity.com/paper/2302.11070/) (2023)
- [Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policies](https://mlsanity.com/paper/2310.16686/) (2023)
- [Recurrent Hypernetworks are Surprisingly Strong in Meta-RL](https://mlsanity.com/paper/2309.14970/) (2023)
- [Discovering General Reinforcement Learning Algorithms with Adversarial Environment Design](https://mlsanity.com/paper/2310.02782/) (2023)
- [Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusion](https://mlsanity.com/paper/2407.10973/) (2024)
- [HyperMARL: Adaptive Hypernetworks for Multi-Agent RL](https://mlsanity.com/paper/2412.04233/) (2024)
- [A Simple Yet Effective Strategy to Robustify the Meta Learning Paradigm](https://mlsanity.com/paper/2310.00708/) (2023)
- [Learning Pareto Set for Multi-Objective Continuous Robot Control](https://mlsanity.com/paper/2406.18924/) (2024)
- [Partial Hypernetworks for Continual Learning](https://mlsanity.com/paper/2306.10724/) (2023)
- [HyperVLA: Efficient Inference in Vision-Language-Action Models via Hypernetworks](https://mlsanity.com/paper/2510.04898/) (2025)
- [StreamAdapter: Efficient Test Time Adaptation from Contextual Streams](https://mlsanity.com/paper/2411.09289/) (2024)
- [Theoretical Investigations and Practical Enhancements on Tail Task Risk Minimization in Meta Learning](https://mlsanity.com/paper/2410.22788/) (2024)
- [Hyper-GoalNet: Goal-Conditioned Manipulation Policy Learning with HyperNetworks](https://mlsanity.com/paper/2512.00085/) (2025)
- [Personalizing Federated Instrument Segmentation with Visual Trait Priors in Robotic Surgery](https://mlsanity.com/paper/2408.03208/) (2024)
- [HyperCLIP: Adapting Vision-Language models with Hypernetworks](https://mlsanity.com/paper/2412.16777/) (2024)
- [HypeRL: Hypernetwork-Based Reinforcement Learning for Control of Parametrized Dynamical Systems](https://mlsanity.com/paper/2501.04538/) (2025)
- [Capability-Aware Shared Hypernetworks for Flexible Heterogeneous Multi-Robot Coordination](https://mlsanity.com/paper/2501.06058/) (2025)
- [Flow to Learn: Flow Matching on Neural Network Parameters](https://mlsanity.com/paper/2503.19371/) (2025)
- [SplAgger: Split Aggregation for Meta-Reinforcement Learning](https://mlsanity.com/paper/2403.03020/) (2024)
- [Enhancing Diversity in Bayesian Deep Learning via Hyperspherical Energy Minimization of CKA](https://mlsanity.com/paper/2411.00259/) (2024)
- [Uncertainty quantification of neural network models of evolving processes via Langevin sampling](https://mlsanity.com/paper/2504.14854/) (2025)
- [Flows and Diffusions on the Neural Manifold](https://mlsanity.com/paper/2507.10623/) (2025)
- [HypeMARL: Multi-Agent Reinforcement Learning For High-Dimensional, Parametric, and Distributed Systems](https://mlsanity.com/paper/2509.16709/) (2025)
- [Robotic Policy Adaptation via Weight-Space Meta-Learning](https://mlsanity.com/paper/2606.07217/) (2026)
- [Deep Reinforcement Learning with Task-Adaptive Retrieval via Hypernetwork](https://mlsanity.com/paper/2306.10698/) (2023)
- [RL3: Boosting Meta Reinforcement Learning via RL inside RL2](https://mlsanity.com/paper/2306.15909/) (2023)
- [Neuromodulated Meta-Learning](https://mlsanity.com/paper/2411.06746/) (2024)
- [Context Representation via Action-Free Transformer encoder-decoder for Meta Reinforcement Learning](https://mlsanity.com/paper/2512.14057/) (2025)

## Abstract (arXiv)

Training a reinforcement learning (RL) agent on a real-world robotics task remains generally impractical due to sample inefficiency. Multi-task RL and meta-RL aim to improve sample efficiency by generalizing over a distribution of related tasks. However, doing so is difficult in practice: In multi-task RL, state of the art methods often fail to outperform a degenerate solution that simply learns each task separately. Hypernetworks are a promising path forward since they replicate the separate policies of the degenerate solution while also allowing for generalization across tasks, and are applicable to meta-RL. However, evidence from supervised learning suggests hypernetwork performance is highly sensitive to the initialization. In this paper, we 1) show that hypernetwork initialization is also a critical factor in meta-RL, and that naive initializations yield poor performance; 2) propose a novel hypernetwork initialization scheme that matches or exceeds the performance of a state-of-the-art approach proposed for supervised settings, as well as being simpler and more general; and 3) use this method to show that hypernetworks can improve performance in meta-RL by evaluating on multiple simulated robotics benchmarks.