# Playing Atari with Six Neurons

arXiv: https://arxiv.org/abs/1806.01363 · HTML: https://mlsanity.com/paper/1806.01363/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2018.

## Built on by (most cited first, 11 of 11)

- [Multi-Game Decision Transformers](https://mlsanity.com/paper/2205.15241/) (2022)
- [On Transforming Reinforcement Learning by Transformer: The Development Trajectory](https://mlsanity.com/paper/2212.14164/) (2022)
- [Learning Belief Representations for Imitation Learning in POMDPs](https://mlsanity.com/paper/1906.09510/) (2019)
- [Representation Learning for Event-based Visuomotor Policies](https://mlsanity.com/paper/2103.00806/) (2021)
- [Learning to Reason in 13 Parameters](https://mlsanity.com/paper/2602.04118/) (2026)
- [Improved Cooperation by Exploiting a Common Signal](https://mlsanity.com/paper/2102.02304/) (2021)
- [Solving Deep Reinforcement Learning Tasks with Evolution Strategies and Linear Policy Networks](https://mlsanity.com/paper/2402.06912/) (2024)
- [Visual processing in context of reinforcement learning](https://mlsanity.com/paper/2208.12525/) (2022)
- [Evolutionary Machine Learning and Games](https://mlsanity.com/paper/2311.16172/) (2023)
- [Smart Transportation Without Neurons -- Fair Metro Network Expansion with Tabular Reinforcement Learning](https://mlsanity.com/paper/2606.04167/) (2026)
- [Emergent Sparsity in Frozen Random CNN Feature Extractors for Deep Reinforcement Learning](https://mlsanity.com/paper/2607.26059/) (2026)

## Abstract (arXiv)

Deep reinforcement learning, applied to vision-based problems like Atari games, maps pixels directly to actions; internally, the deep neural network bears the responsibility of both extracting useful information and making decisions based on it. By separating the image processing from decision-making, one could better understand the complexity of each task, as well as potentially find smaller policy representations that are easier for humans to understand and may generalize better. To this end, we propose a new method for learning policies and compact state representations separately but simultaneously for policy approximation in reinforcement learning. State representations are generated by an encoder based on two novel algorithms: Increasing Dictionary Vector Quantization makes the encoder capable of growing its dictionary size over time, to address new observations as they appear in an open-ended online-learning context; Direct Residuals Sparse Coding encodes observations by disregarding reconstruction error minimization, and aiming instead for highest information inclusion. The encoder autonomously selects observations online to train on, in order to maximize code sparsity. As the dictionary size increases, the encoder produces increasingly larger inputs for the neural network: this is addressed by a variation of the Exponential Natural Evolution Strategies algorithm which adapts its probability distribution dimensionality along the run. We test our system on a selection of Atari games using tiny neural networks of only 6 to 18 neurons (depending on the game's controls). These are still capable of achieving results comparable---and occasionally superior---to state-of-the-art techniques which use two orders of magnitude more neurons.