# Back to Basics: Benchmarking Canonical Evolution Strategies for Playing Atari

arXiv: https://arxiv.org/abs/1802.08842 · HTML: https://mlsanity.com/paper/1802.08842/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2018.

## Built on by (most cited first, 28 of 28)

- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Automated Dynamic Algorithm Configuration](https://mlsanity.com/paper/2205.13881/) (2022)
- [Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning](https://mlsanity.com/paper/2509.24372/) (2025)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey](https://mlsanity.com/paper/2110.01411/) (2021)
- [Combining Evolution and Deep Reinforcement Learning for Policy Search: a Survey](https://mlsanity.com/paper/2203.14009/) (2022)
- [Deep Black-Box Reinforcement Learning with Movement Primitives](https://mlsanity.com/paper/2210.09622/) (2022)
- [Derivative-Free Reinforcement Learning: A Review](https://mlsanity.com/paper/2102.05710/) (2021)
- [Unlocking the Power of Representations in Long-term Novelty-based Exploration](https://mlsanity.com/paper/2305.01521/) (2023)
- [MP3: Movement Primitive-Based (Re-)Planning Policy](https://mlsanity.com/paper/2306.12729/) (2023)
- [Understanding and Avoiding AI Failures: A Practical Guide](https://mlsanity.com/paper/2104.12582/) (2021)
- [ESSA: Evolutionary Strategies for Scalable Alignment](https://mlsanity.com/paper/2507.04453/) (2025)
- [Rethinking Population-assisted Off-policy Reinforcement Learning](https://mlsanity.com/paper/2305.02949/) (2023)
- [Evolutionary Policy Optimization](https://mlsanity.com/paper/2503.19037/) (2025)
- [Multitask Neuroevolution for Reinforcement Learning with Long and Short Episodes](https://mlsanity.com/paper/2203.10844/) (2022)
- [The Factory Must Grow: Automation in Factorio](https://mlsanity.com/paper/2102.04871/) (2021)
- [Solving Deep Reinforcement Learning Tasks with Evolution Strategies and Linear Policy Networks](https://mlsanity.com/paper/2402.06912/) (2024)
- [Explicit and Non-asymptotic Query Complexities of Rank-Based Zeroth-order Algorithms on Smooth Functions](https://mlsanity.com/paper/2512.16200/) (2025)
- [Potential Impacts of Smart Homes on Human Behavior: A Reinforcement Learning Approach](https://mlsanity.com/paper/2102.13307/) (2021)
- [Guiding Evolutionary Strategies by Differentiable Robot Simulators](https://mlsanity.com/paper/2110.00438/) (2021)
- [Evasive Active Hypothesis Testing with Deep Neuroevolution: The Single- and Multi-Agent Cases](https://mlsanity.com/paper/2403.10112/) (2024)
- [EvoRL: A GPU-accelerated Framework for Evolutionary Reinforcement Learning](https://mlsanity.com/paper/2501.15129/) (2025)
- [Surrogate-Assisted Evolutionary Reinforcement Learning Based on Autoencoder and Hyperbolic Neural Network](https://mlsanity.com/paper/2505.19423/) (2025)
- [Curiosity creates Diversity in Policy Search](https://mlsanity.com/paper/2212.03530/) (2022)
- [Evolutionary Machine Learning and Games](https://mlsanity.com/paper/2311.16172/) (2023)
- [Random Search as a Baseline for Sparse Neural Network Architecture Search](https://mlsanity.com/paper/2403.08265/) (2024)
- [Active Sensing for RIS-Aided Tracking and Power Control: A Hybrid Neuroevolution and Supervised Learning Approach](https://mlsanity.com/paper/2607.00056/) (2026)
- [Learning to Play: Reinforcement Learning and Games](https://mlsanity.com/paper/2610.17391/) (2026)

## Abstract (arXiv)

Evolution Strategies (ES) have recently been demonstrated to be a viable alternative to reinforcement learning (RL) algorithms on a set of challenging deep RL problems, including Atari games and MuJoCo humanoid locomotion benchmarks. While the ES algorithms in that work belonged to the specialized class of natural evolution strategies (which resemble approximate gradient RL algorithms, such as REINFORCE), we demonstrate that even a very basic canonical ES algorithm can achieve the same or even better performance. This success of a basic ES algorithm suggests that the state-of-the-art can be advanced further by integrating the many advances made in the field of ES in the last decades. We also demonstrate qualitatively that ES algorithms have very different performance characteristics than traditional RL algorithms: on some games, they learn to exploit the environment and perform much better while on others they can get stuck in suboptimal local minima. Combining their strengths with those of traditional RL algorithms is therefore likely to lead to new advances in the state of the art.