# The Utility of Sparse Representations for Control in Reinforcement Learning

arXiv: https://arxiv.org/abs/1811.06626 · HTML: https://mlsanity.com/paper/1811.06626/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2018.

## Built on by (most cited first, 24 of 24)

- [Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning](https://mlsanity.com/paper/2404.00781/) (2024)
- [Streaming Deep Reinforcement Learning Finally Works](https://mlsanity.com/paper/2410.14606/) (2024)
- [Investigating the Properties of Neural Network Representations in Reinforcement Learning](https://mlsanity.com/paper/2203.15955/) (2022)
- [Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillation](https://mlsanity.com/paper/2109.00525/) (2021)
- [Correcting Momentum in Temporal Difference Learning](https://mlsanity.com/paper/2106.03955/) (2021)
- [SPeCiaL: Self-Supervised Pretraining for Continual Learning](https://mlsanity.com/paper/2106.09065/) (2021)
- [Locality Sensitive Sparse Encoding for Learning World Models Online](https://mlsanity.com/paper/2401.13034/) (2024)
- [Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents](https://mlsanity.com/paper/2510.13704/) (2025)
- [Elephant Neural Networks: Born to Be a Continual Learner](https://mlsanity.com/paper/2310.01365/) (2023)
- [Online Reinforcement Learning in Non-Stationary Context-Driven Environments](https://mlsanity.com/paper/2302.02182/) (2023)
- [Utility-based Perturbed Gradient Descent: An Optimizer for Continual Learning](https://mlsanity.com/paper/2302.03281/) (2023)
- [Measuring and Mitigating Interference in Reinforcement Learning](https://mlsanity.com/paper/2307.04887/) (2023)
- [Parameter-free Gradient Temporal Difference Learning](https://mlsanity.com/paper/2105.04129/) (2021)
- [The Impact of Data Distribution on Q-learning with Function Approximation](https://mlsanity.com/paper/2111.11758/) (2021)
- [Tackling Visual Control via Multi-View Exploration Maximization](https://mlsanity.com/paper/2211.15233/) (2022)
- [Hadamard Representation: Scaffolding Performance Across Model-free RL](https://mlsanity.com/paper/2406.09079/) (2024)
- [Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learning](https://mlsanity.com/paper/2602.12375/) (2026)
- [Knowledge Consolidation based Class Incremental Online Learning with Limited Data](https://mlsanity.com/paper/2106.06795/) (2021)
- [Online Attentive Kernel-Based Temporal Difference Learning](https://mlsanity.com/paper/2201.09065/) (2022)
- [Visual Radial Basis Q-Network](https://mlsanity.com/paper/2206.06712/) (2022)
- [Optimal Control with Natural Images: Efficient Reinforcement Learning using Overcomplete Sparse Codes](https://mlsanity.com/paper/2412.08893/) (2024)
- [Efficient Reinforcement Learning by Reducing Forgetting with Elephant Activation Functions](https://mlsanity.com/paper/2509.19159/) (2025)
- [Learning Distinguishable Representations in Deep Q-Networks for Linear Transfer](https://mlsanity.com/paper/2509.24947/) (2025)
- [Sparse-TD-JEPA: Improving Zero-Shot Reinforcement Learning With Sparse Latent Task Representations](https://mlsanity.com/paper/2610.14400/) (2026)

## Abstract (arXiv)

We investigate sparse representations for control in reinforcement learning. While these representations are widely used in computer vision, their prevalence in reinforcement learning is limited to sparse coding where extracting representations for new data can be computationally intensive. Here, we begin by demonstrating that learning a control policy incrementally with a representation from a standard neural network fails in classic control domains, whereas learning with a representation obtained from a neural network that has sparsity properties enforced is effective. We provide evidence that the reason for this is that the sparse representation provides locality, and so avoids catastrophic interference, and particularly keeps consistent, stable values for bootstrapping. We then discuss how to learn such sparse representations. We explore the idea of Distributional Regularizers, where the activation of hidden nodes is encouraged to match a particular distribution that results in sparse activation across time. We identify a simple but effective way to obtain sparse representations, not afforded by previously proposed strategies, making it more practical for further investigation into sparse representations for reinforcement learning.