# On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2105.01648 · HTML: https://mlsanity.com/paper/2105.01648/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 4 May 2021.

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [The Lottery Ticket Hypothesis for Pre-trained BERT Networks](https://mlsanity.com/paper/2007.12223/) (2020)
- [The Early Phase of Neural Network Training](https://mlsanity.com/paper/2002.10365/) (2020)
- [One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers](https://mlsanity.com/paper/1906.02773/) (2019)
- [The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models](https://mlsanity.com/paper/2012.06908/) (2020)
- [Kickstarting Deep Reinforcement Learning](https://mlsanity.com/paper/1803.03835/) (2018)
- [Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research](https://mlsanity.com/paper/2011.14826/) (2020)
- [Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP](https://mlsanity.com/paper/1906.02768/) (2019)
- [rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch](https://mlsanity.com/paper/1909.01500/) (2019)
- [The Lottery Ticket Hypothesis for Object Recognition](https://mlsanity.com/paper/2012.04643/) (2020)
- [Sparse Transfer Learning via Winning Lottery Tickets](https://mlsanity.com/paper/1905.07785/) (2019)
- [Evaluating Lottery Tickets Under Distributional Shifts](https://mlsanity.com/paper/1910.12708/) (2019)

## Built on by (most cited first, 24 of 24)

- [The State of Sparse Training in Deep Reinforcement Learning](https://mlsanity.com/paper/2206.10369/) (2022)
- [Dynamic Sparse Training for Deep Reinforcement Learning](https://mlsanity.com/paper/2106.04217/) (2021)
- [RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratch](https://mlsanity.com/paper/2205.15043/) (2022)
- [Exploring Lottery Ticket Hypothesis in Spiking Neural Networks](https://mlsanity.com/paper/2207.01382/) (2022)
- [A Survey of Lottery Ticket Hypothesis](https://mlsanity.com/paper/2403.04861/) (2024)
- [Rethinking Graph Lottery Tickets: Graph Sparsity Matters](https://mlsanity.com/paper/2305.02190/) (2023)
- [Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks](https://mlsanity.com/paper/2206.01278/) (2022)
- [Adversarial Cheap Talk](https://mlsanity.com/paper/2211.11030/) (2022)
- [Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learning](https://mlsanity.com/paper/2302.06548/) (2023)
- [Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey](https://mlsanity.com/paper/2205.08099/) (2022)
- [Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning](https://mlsanity.com/paper/2506.17204/) (2025)
- [Differentiable Transportation Pruning](https://mlsanity.com/paper/2307.08483/) (2023)
- [Discovering Minimal Reinforcement Learning Environments](https://mlsanity.com/paper/2406.12589/) (2024)
- [The Multiple Ticket Hypothesis: Random Sparse Subnetworks Suffice for RLVR](https://mlsanity.com/paper/2602.01599/) (2026)
- [Lottery Tickets in Evolutionary Optimization: On Sparse Backpropagation-Free Trainability](https://mlsanity.com/paper/2306.00045/) (2023)
- [Learning Activation Functions for Sparse Neural Networks](https://mlsanity.com/paper/2305.10964/) (2023)
- [Value-Based Deep Multi-Agent Reinforcement Learning with Dynamic Sparse Training](https://mlsanity.com/paper/2409.19391/) (2024)
- [On How Iterative Magnitude Pruning Discovers Local Receptive Fields in Fully Connected Neural Networks](https://mlsanity.com/paper/2412.06545/) (2024)
- [Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learning](https://mlsanity.com/paper/2510.12096/) (2025)
- [Neural Network Optimization for Reinforcement Learning Tasks Using Sparse Computations](https://mlsanity.com/paper/2201.02571/) (2022)
- [Learning Emergence of Interaction Patterns across Independent RL Agents in Multi-Agent Environments](https://mlsanity.com/paper/2410.02516/) (2024)
- [RNM-TD3: N:M Semi-structured Sparse Reinforcement Learning From Scratch](https://mlsanity.com/paper/2602.14578/) (2026)
- [Toy Combinatorial Interpretability Models Reveal Lottery Tickets in Early Feature Space](https://mlsanity.com/paper/2605.17704/) (2026)
- [Modeling and Understanding Neural Weights of Humanoid Policies with Diffusion](https://mlsanity.com/paper/2610.20606/) (2026)

## Abstract (arXiv)

The lottery ticket hypothesis questions the role of overparameterization in supervised deep learning. But how is the performance of winning lottery tickets affected by the distributional shift inherent to reinforcement learning problems? In this work, we address this question by comparing sparse agents who have to address the non-stationarity of the exploration-exploitation problem with supervised agents trained to imitate an expert. We show that feed-forward networks trained with behavioural cloning compared to reinforcement learning can be pruned to higher levels of sparsity without performance degradation. This suggests that in order to solve the RL-specific distributional shift agents require more degrees of freedom. Using a set of carefully designed baseline conditions, we find that the majority of the lottery ticket effect in both learning paradigms can be attributed to the identified mask rather than the weight initialization. The input layer mask selectively prunes entire input dimensions that turn out to be irrelevant for the task at hand. At a moderate level of sparsity the mask identified by iterative magnitude pruning yields minimal task-relevant representations, i.e., an interpretable inductive bias. Finally, we propose a simple initialization rescaling which promotes the robust identification of sparse task representations in low-dimensional control tasks.