# Asynchronous Evolution of Deep Neural Network Architectures

arXiv: https://arxiv.org/abs/2308.04102 · HTML: https://mlsanity.com/paper/2308.04102/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 61 references that are in the map.
- First posted on arXiv 8 Aug 2023.

## Builds on (most cited first, 24 of 24)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Scalable Bayesian Optimization Using Deep Neural Networks](https://mlsanity.com/paper/1502.05700/) (2015)
- [Image Captioning with Semantic Attention](https://mlsanity.com/paper/1603.03925/) (2016)
- [A Comprehensive Survey of Deep Learning for Image Captioning](https://mlsanity.com/paper/1810.04020/) (2018)
- [Evolving Deep Neural Networks](https://mlsanity.com/paper/1703.00548/) (2017)
- [CMA-ES for Hyperparameter Optimization of Deep Neural Networks](https://mlsanity.com/paper/1604.07269/) (2016)
- [Context-aware Captions from Context-agnostic Supervision](https://mlsanity.com/paper/1701.02870/) (2017)
- [Evolutionary Architecture Search For Deep Multitask Networks](https://mlsanity.com/paper/1803.03745/) (2018)
- [Evolutionary Neural AutoML for Deep Learning](https://mlsanity.com/paper/1902.06827/) (2019)
- [NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm](https://mlsanity.com/paper/1810.03522/) (2018)
- [Hyperparameter Tuning for Deep Reinforcement Learning Applications](https://mlsanity.com/paper/2201.11182/) (2022)
- [Regularized Evolutionary Population-Based Training](https://mlsanity.com/paper/2002.04225/) (2020)
- [Enhanced Optimization with Composite Objectives and Novelty Selection](https://mlsanity.com/paper/1803.03744/) (2018)
- [Enhanced Optimization with Composite Objectives and Novelty Pulsation](https://mlsanity.com/paper/1906.04050/) (2019)
- [The Impact of Asynchrony on Parallel Model-Based EAs](https://mlsanity.com/paper/2303.15543/) (2023)

## Abstract (arXiv)

Many evolutionary algorithms (EAs) take advantage of parallel evaluation of candidates. However, if evaluation times vary significantly, many worker nodes (i.e., compute clients) are idle much of the time, waiting for the next generation to be created. Evolutionary neural architecture search (ENAS), a class of EAs that optimizes the architecture and hyperparameters of deep neural networks, is particularly vulnerable to this issue. This paper proposes a generic asynchronous evaluation strategy (AES) that is then adapted to work with ENAS. AES increases throughput by maintaining a queue of up to K individuals ready to be sent to the workers for evaluation and proceeding to the next generation as soon as M<<K individuals have been evaluated. A suitable value for M is determined experimentally, balancing diversity and efficiency. To showcase the generality and power of AES, it was first evaluated in eight-line sorting network design (a single-population optimization task with limited evaluation-time variability), achieving an over two-fold speedup. Next, it was evaluated in 11-bit multiplexer design (a single-population discovery task with extended variability), where a 14-fold speedup was observed. It was then scaled up to ENAS for image captioning (a multi-population open-ended-optimization task), resulting in an over two-fold speedup. In all problems, a multifold performance improvement was observed, suggesting that AES is a promising method for parallelizing the evolution of complex systems with long and variable evaluation times, such as those in ENAS.