# Hypernetworks for Zero-shot Transfer in Reinforcement Learning

arXiv: https://arxiv.org/abs/2211.15457 · HTML: https://mlsanity.com/paper/2211.15457/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 67 references that are in the map.
- First posted on arXiv 28 Nov 2022.

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [A Survey on Deep Transfer Learning](https://mlsanity.com/paper/1808.01974/) (2018)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Contextual Decision Processes with Low Bellman Rank are PAC-Learnable](https://mlsanity.com/paper/1610.09512/) (2016)
- [Continual learning with hypernetworks](https://mlsanity.com/paper/1906.00695/) (2019)
- [Distral: Robust Multitask Reinforcement Learning](https://mlsanity.com/paper/1707.04175/) (2017)
- [Contextual Markov Decision Processes](https://mlsanity.com/paper/1502.02259/) (2015)
- [Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble](https://mlsanity.com/paper/2107.00591/) (2021)
- [RUDDER: Return Decomposition for Delayed Rewards](https://mlsanity.com/paper/1806.07857/) (2018)
- [Meta-Gradient Reinforcement Learning](https://mlsanity.com/paper/1805.09801/) (2018)
- [Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions](https://mlsanity.com/paper/1912.02875/) (2019)
- [Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments](https://mlsanity.com/paper/1910.07224/) (2019)
- [Training Agents using Upside-Down Reinforcement Learning](https://mlsanity.com/paper/1912.02877/) (2019)
- [Universal Successor Features Approximators](https://mlsanity.com/paper/1812.07626/) (2018)
- [On Learning Intrinsic Rewards for Policy Gradient Methods](https://mlsanity.com/paper/1804.06459/) (2018)
- [Hindsight Credit Assignment](https://mlsanity.com/paper/1912.02503/) (2019)
- [learn2learn: A Library for Meta-Learning Research](https://mlsanity.com/paper/2008.12284/) (2020)
- [Bayesian Hypernetworks](https://mlsanity.com/paper/1710.04759/) (2017)

## Built on by (most cited first, 19 of 19)

- [A Brief Review of Hypernetworks in Deep Learning](https://mlsanity.com/paper/2306.06955/) (2023)
- [Universal Morphology Control via Contextual Modulation](https://mlsanity.com/paper/2302.11070/) (2023)
- [Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusion](https://mlsanity.com/paper/2407.10973/) (2024)
- [Distilling Morphology-Conditioned Hypernetworks for Efficient Universal Morphology Control](https://mlsanity.com/paper/2402.06570/) (2024)
- [Learning Pareto Set for Multi-Objective Continuous Robot Control](https://mlsanity.com/paper/2406.18924/) (2024)
- [Model-Based Transfer Learning for Contextual Reinforcement Learning](https://mlsanity.com/paper/2408.04498/) (2024)
- [HyperVLA: Efficient Inference in Vision-Language-Action Models via Hypernetworks](https://mlsanity.com/paper/2510.04898/) (2025)
- [Hyper-GoalNet: Goal-Conditioned Manipulation Policy Learning with HyperNetworks](https://mlsanity.com/paper/2512.00085/) (2025)
- [Episodic Reinforcement Learning with Expanded State-reward Space](https://mlsanity.com/paper/2401.10516/) (2024)
- [Learning to Look: Seeking Information for Decision Making via Policy Factorization](https://mlsanity.com/paper/2410.18964/) (2024)
- [Capability-Aware Shared Hypernetworks for Flexible Heterogeneous Multi-Robot Coordination](https://mlsanity.com/paper/2501.06058/) (2025)
- [Adaptive Shielding for Safe Reinforcement Learning under Hidden-Parameter Dynamics Shifts](https://mlsanity.com/paper/2506.11033/) (2025)
- [Domain Generalization Deep Graph Transformation](https://mlsanity.com/paper/2305.11389/) (2023)
- [TRAM: Test-Time Risk Adaptation with Mixture of Agents](https://mlsanity.com/paper/2408.08812/) (2024)
- [DCoPilot: Generative AI-Empowered Policy Adaptation for Dynamic Data Center Operations](https://mlsanity.com/paper/2602.02137/) (2026)
- [TeNet: Text-to-Network for Compact Policy Synthesis](https://mlsanity.com/paper/2601.15912/) (2026)
- [Temporal Consistency Improves Generalization in Contextual Offline Meta Reinforcement Learning](https://mlsanity.com/paper/2603.02935/) (2026)
- [Amortized Low-Rank Adaptation for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2609.12278/) (2026)
- [MOCHA: Multi-Objective Co-Design using Hypernetwork Architectures](https://mlsanity.com/paper/2609.30570/) (2026)

## Abstract (arXiv)

In this paper, hypernetworks are trained to generate behaviors across a range of unseen task conditions, via a novel TD-based training objective and data from a set of near-optimal RL solutions for training tasks. This work relates to meta RL, contextual RL, and transfer learning, with a particular focus on zero-shot performance at test time, enabled by knowledge of the task parameters (also known as context). Our technical approach is based upon viewing each RL algorithm as a mapping from the MDP specifics to the near-optimal value function and policy and seek to approximate it with a hypernetwork that can generate near-optimal value functions and policies, given the parameters of the MDP. We show that, under certain conditions, this mapping can be considered as a supervised learning problem. We empirically evaluate the effectiveness of our method for zero-shot transfer to new reward and transition dynamics on a series of continuous control tasks from DeepMind Control Suite. Our method demonstrates significant improvements over baselines from multitask and meta RL approaches.