# Learning Sparse Representations in Reinforcement Learning with Sparse Coding

arXiv: https://arxiv.org/abs/1707.08316 · HTML: https://mlsanity.com/paper/1707.08316/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2017.

## Built on by (most cited first, 5 of 5)

- [Elephant Neural Networks: Born to Be a Continual Learner](https://mlsanity.com/paper/2310.01365/) (2023)
- [Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learning](https://mlsanity.com/paper/2602.12375/) (2026)
- [Optimal Control with Natural Images: Efficient Reinforcement Learning using Overcomplete Sparse Codes](https://mlsanity.com/paper/2412.08893/) (2024)
- [Efficient Reinforcement Learning by Reducing Forgetting with Elephant Activation Functions](https://mlsanity.com/paper/2509.19159/) (2025)
- [Ultrafast On-Chip Online Learning via Spline Locality in Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2602.02056/) (2026)

## Abstract (arXiv)

A variety of representation learning approaches have been investigated for reinforcement learning; much less attention, however, has been given to investigating the utility of sparse coding. Outside of reinforcement learning, sparse coding representations have been widely used, with non-convex objectives that result in discriminative representations. In this work, we develop a supervised sparse coding objective for policy evaluation. Despite the non-convexity of this objective, we prove that all local minima are global minima, making the approach amenable to simple optimization strategies. We empirically show that it is key to use a supervised objective, rather than the more straightforward unsupervised sparse coding approach. We compare the learned representations to a canonical fixed sparse representation, called tile-coding, demonstrating that the sparse coding representation outperforms a wide variety of tilecoding representations.