# On the Relationship Between the OpenAI Evolution Strategy and Stochastic Gradient Descent

arXiv: https://arxiv.org/abs/1712.06564 · HTML: https://mlsanity.com/paper/1712.06564/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Dec 2017.

## Built on by (most cited first, 16 of 16)

- [Deep Multiagent Reinforcement Learning: Challenges and Directions](https://mlsanity.com/paper/2106.15691/) (2021)
- [Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning](https://mlsanity.com/paper/2509.24372/) (2025)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [Brain-inspired learning in artificial neural networks: a review](https://mlsanity.com/paper/2305.11252/) (2023)
- [Derivative-Free Reinforcement Learning: A Review](https://mlsanity.com/paper/2102.05710/) (2021)
- [Pretrained Optimization Model for Zero-Shot Black Box Optimization](https://mlsanity.com/paper/2405.03728/) (2024)
- [EvoTorch: Scalable Evolutionary Computation in Python](https://mlsanity.com/paper/2302.12600/) (2023)
- [Evolutionary Strategies lead to Catastrophic Forgetting in LLMs](https://mlsanity.com/paper/2601.20861/) (2026)
- [RankNEAT: Outperforming Stochastic Gradient Search in Preference Learning Tasks](https://mlsanity.com/paper/2204.06901/) (2022)
- [Direct evaluation of progression or regression of disease burden in brain metastatic disease with Deep Neuroevolution](https://mlsanity.com/paper/2203.12853/) (2022)
- [Learning Discrete Structured Variational Auto-Encoder using Natural Evolution Strategies](https://mlsanity.com/paper/2205.01324/) (2022)
- [Unsupervised Discovery of Inertial-Fusion Plasma Physics using Differentiable Kinetic Simulations and a Maximum Entropy Loss Function](https://mlsanity.com/paper/2206.01637/) (2022)
- [Enabling Incremental Training with Forward Pass for Edge Devices](https://mlsanity.com/paper/2103.14007/) (2021)
- [Biologically Plausible Training of Deep Neural Networks Using a Top-down Credit Assignment Network](https://mlsanity.com/paper/2208.01416/) (2022)
- [A Unified Perspective on Optimization in Machine Learning and Neuroscience: From Gradient Descent to Neural Adaptation](https://mlsanity.com/paper/2510.18812/) (2025)
- [Knowing in Advance When an Evolutionary Outer Loop Will Not Help: A Pre-Registered Cheap-Baseline Screening Rule](https://mlsanity.com/paper/2606.29119/) (2026)

## Abstract (arXiv)

Because stochastic gradient descent (SGD) has shown promise optimizing neural networks with millions of parameters and few if any alternatives are known to exist, it has moved to the heart of leading approaches to reinforcement learning (RL). For that reason, the recent result from OpenAI showing that a particular kind of evolution strategy (ES) can rival the performance of SGD-based deep RL methods with large neural networks provoked surprise. This result is difficult to interpret in part because of the lingering ambiguity on how ES actually relates to SGD. The aim of this paper is to significantly reduce this ambiguity through a series of MNIST-based experiments designed to uncover their relationship. As a simple supervised problem without domain noise (unlike in most RL), MNIST makes it possible (1) to measure the correlation between gradients computed by ES and SGD and (2) then to develop an SGD-based proxy that accurately predicts the performance of different ES population sizes. These innovations give a new level of insight into the real capabilities of ES, and lead also to some unconventional means for applying ES to supervised problems that shed further light on its differences from SGD. Incorporating these lessons, the paper concludes by demonstrating that ES can achieve 99% accuracy on MNIST, a number higher than any previously published result for any evolutionary method. While not by any means suggesting that ES should substitute for SGD in supervised learning, the suite of experiments herein enables more informed decisions on the application of ES within RL and other paradigms.