# Evolutionary Stochastic Gradient Descent for Optimization of Deep Neural Networks

arXiv: https://arxiv.org/abs/1810.06773 · HTML: https://mlsanity.com/paper/1810.06773/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2018.

## Built on by (most cited first, 13 of 13)

- [Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies](https://mlsanity.com/paper/2112.13835/) (2021)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [PopulAtion Parameter Averaging (PAPA)](https://mlsanity.com/paper/2304.03094/) (2023)
- [Half a Dozen Real-World Applications of Evolutionary Multitasking, and More](https://mlsanity.com/paper/2109.13101/) (2021)
- [Perception-Distortion Balanced Super-Resolution: A Multi-Objective Optimization Perspective](https://mlsanity.com/paper/2312.15408/) (2023)
- [Evolutionary Optimization of Physics-Informed Neural Networks: Evo-PINN Frontiers and Opportunities](https://mlsanity.com/paper/2501.06572/) (2025)
- [Optimizing Neural Networks with Gradient Lexicase Selection](https://mlsanity.com/paper/2312.12606/) (2023)
- [Efficient Personalization of Quantized Diffusion Model without Backpropagation](https://mlsanity.com/paper/2503.14868/) (2025)
- [Convolutional Neural Network(CNN/ConvNet) in Stock Price Movement Prediction](https://mlsanity.com/paper/2106.01920/) (2021)
- [Bridging Models to Defend: A Population-Based Strategy for Robust Adversarial Defense](https://mlsanity.com/paper/2303.10225/) (2023)
- [Extended Linear Regression: A Kalman Filter Approach for Minimizing Loss via Area Under the Curve](https://mlsanity.com/paper/2308.12280/) (2023)
- [Scrap Your Schedules with PopDescent](https://mlsanity.com/paper/2310.14671/) (2023)
- [Ever Evolving Evaluator (EV3): Towards Flexible and Reliable Meta-Optimization for Knowledge Distillation](https://mlsanity.com/paper/2310.18893/) (2023)

## Abstract (arXiv)

We propose a population-based Evolutionary Stochastic Gradient Descent (ESGD) framework for optimizing deep neural networks. ESGD combines SGD and gradient-free evolutionary algorithms as complementary algorithms in one framework in which the optimization alternates between the SGD step and evolution step to improve the average fitness of the population. With a back-off strategy in the SGD step and an elitist strategy in the evolution step, it guarantees that the best fitness in the population will never degrade. In addition, individuals in the population optimized with various SGD-based optimizers using distinct hyper-parameters in the SGD step are considered as competing species in a coevolution setting such that the complementarity of the optimizers is also taken into account. The effectiveness of ESGD is demonstrated across multiple applications including speech recognition, image recognition and language modeling, using networks with a variety of deep architectures.