# On the Interplay Between Sparsity and Training in Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2501.16729 · HTML: https://mlsanity.com/paper/2501.16729/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 28 Jan 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 21 of 21)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [Weight Agnostic Neural Networks](https://mlsanity.com/paper/1906.04358/) (2019)
- [Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers](https://mlsanity.com/paper/2005.06870/) (2020)
- [The State of Sparse Training in Deep Reinforcement Learning](https://mlsanity.com/paper/2206.10369/) (2022)
- [Dynamic Sparse Training for Deep Reinforcement Learning](https://mlsanity.com/paper/2106.04217/) (2021)
- [Simple Agent, Complex Environment: Efficient Reinforcement Learning with Agent States](https://mlsanity.com/paper/2102.05261/) (2021)
- [Dynamic Sparse Network for Time Series Classification: Learning What to "see''](https://mlsanity.com/paper/2212.09840/) (2022)
- [Neural Network Training Using ℓ1-Regularization and Bi-fidelity Data](https://mlsanity.com/paper/2105.13011/) (2021)
- [The Quest for a Common Model of the Intelligent Decision Maker](https://mlsanity.com/paper/2202.13252/) (2022)
- [Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learning](https://mlsanity.com/paper/2302.06548/) (2023)
- [Faster Deep Reinforcement Learning with Slower Online Network](https://mlsanity.com/paper/2112.05848/) (2021)
- [On the Convergence of Bounded Agents](https://mlsanity.com/paper/2307.11044/) (2023)
- [Learning Sparse Representations Incrementally in Deep Reinforcement Learning](https://mlsanity.com/paper/1912.04002/) (2019)
- [Towards model-free RL algorithms that scale well with unstructured data](https://mlsanity.com/paper/2311.02215/) (2023)
- [Adapting the Function Approximation Architecture in Online Reinforcement Learning](https://mlsanity.com/paper/2106.09776/) (2021)
- [Are Sparse Neural Networks Better Hard Sample Learners?](https://mlsanity.com/paper/2409.09196/) (2024)

## Built on by (most cited first, 3 of 3)

- [Reinforcement Learning Finetunes Small Subnetworks in Large Language Models](https://mlsanity.com/paper/2505.11711/) (2025)
- [Reinforcement Learning Fine-Tunes a Sparse Subnetwork in Large Language Models](https://mlsanity.com/paper/2507.17107/) (2025)
- [Emergent Sparsity in Frozen Random CNN Feature Extractors for Deep Reinforcement Learning](https://mlsanity.com/paper/2607.26059/) (2026)

## Abstract (arXiv)

We study the benefits of different sparse architectures for deep reinforcement learning. In particular, we focus on image-based domains where spatially-biased and fully-connected architectures are common. Using these and several other architectures of equal capacity, we show that sparse structure has a significant effect on learning performance. We also observe that choosing the best sparse architecture for a given domain depends on whether the hidden layer weights are fixed or learned.