# Goal-Conditioned Generators of Deep Policies

arXiv: https://arxiv.org/abs/2207.01570 · HTML: https://mlsanity.com/paper/2207.01570/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 51 references that are in the map.
- First posted on arXiv 4 Jul 2022.

## Builds on (most cited first, 24 of 24)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [Continual learning with hypernetworks](https://mlsanity.com/paper/1906.00695/) (2019)
- [Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions](https://mlsanity.com/paper/1912.02875/) (2019)
- [Training Agents using Upside-Down Reinforcement Learning](https://mlsanity.com/paper/1912.02877/) (2019)
- [Parameter Prediction for Unseen Deep Architectures](https://mlsanity.com/paper/2110.13100/) (2021)
- [Differentiable plasticity: training plastic neural networks with backpropagation](https://mlsanity.com/paper/1804.02464/) (2018)
- [On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models](https://mlsanity.com/paper/1511.09249/) (2015)
- [Meta Learning Backpropagation And Improving It](https://mlsanity.com/paper/2012.14905/) (2020)
- [Meta-Learning through Hebbian Plasticity in Random Networks](https://mlsanity.com/paper/2007.02686/) (2020)
- [A Modern Self-Referential Weight Matrix That Learns to Modify Itself](https://mlsanity.com/paper/2202.05780/) (2022)
- [Hindsight policy gradients](https://mlsanity.com/paper/1711.06006/) (2017)
- [Policy Evaluation Networks](https://mlsanity.com/paper/2002.11833/) (2020)
- [Introducing Symmetries to Black Box Meta Reinforcement Learning](https://mlsanity.com/paper/2109.10781/) (2021)
- [Parameter-Based Value Functions](https://mlsanity.com/paper/2006.09226/) (2020)
- [Upside-Down Reinforcement Learning Can Diverge in Stochastic Environments With Episodic Resets](https://mlsanity.com/paper/2205.06595/) (2022)
- [Finding online neural update rules by learning to remember](https://mlsanity.com/paper/2003.03124/) (2020)

## Built on by (most cited first, 9 of 9)

- [Hypernetworks for Zero-shot Transfer in Reinforcement Learning](https://mlsanity.com/paper/2211.15457/) (2022)
- [Learning Useful Representations of Recurrent Neural Network Weight Matrices](https://mlsanity.com/paper/2403.11998/) (2024)
- [Learning to Identify Critical States for Reinforcement Learning from Videos](https://mlsanity.com/paper/2308.07795/) (2023)
- [HyperVLA: Efficient Inference in Vision-Language-Action Models via Hypernetworks](https://mlsanity.com/paper/2510.04898/) (2025)
- [Skill Decision Transformer](https://mlsanity.com/paper/2301.13573/) (2023)
- [Learning One Abstract Bit at a Time Through Self-Invented Experiments Encoded as Neural Networks](https://mlsanity.com/paper/2212.14374/) (2022)
- [From Parameters to Behaviors: Unsupervised Compression of the Policy Space](https://mlsanity.com/paper/2509.22566/) (2025)
- [Upside Down Reinforcement Learning with Policy Generators](https://mlsanity.com/paper/2501.16288/) (2025)
- [Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching](https://mlsanity.com/paper/2603.27044/) (2026)

## Abstract (arXiv)

Goal-conditioned Reinforcement Learning (RL) aims at learning optimal policies, given goals encoded in special command inputs. Here we study goal-conditioned neural nets (NNs) that learn to generate deep NN policies in form of context-specific weight matrices, similar to Fast Weight Programmers and other methods from the 1990s. Using context commands of the form "generate a policy that achieves a desired expected return," our NN generators combine powerful exploration of parameter space with generalization across commands to iteratively find better and better policies. A form of weight-sharing HyperNetworks and policy embeddings scales our method to generate deep NNs. Experiments show how a single learned policy generator can produce policies that achieve any return seen during training. Finally, we evaluate our algorithm on a set of continuous control tasks where it exhibits competitive performance. Our code is public.