# Evolution of Activation Functions for Deep Learning-Based Image Classification

arXiv: https://arxiv.org/abs/2206.12089 · HTML: https://mlsanity.com/paper/2206.12089/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jun 2022.

## Built on by (most cited first, 6 of 6)

- [How important are activation functions in regression and classification? A survey, performance comparison, and future directions](https://mlsanity.com/paper/2209.02681/) (2022)
- [An Evolutionary, Gradient-Free, Query-Efficient, Black-Box Algorithm for Generating Adversarial Instances in Deep Networks](https://mlsanity.com/paper/2208.08297/) (2022)
- [EC-KitY: Evolutionary Computation Tool Kit in Python with Seamless Machine Learning Integration](https://mlsanity.com/paper/2207.10367/) (2022)
- [Activation Function Optimization Scheme for Image Classification](https://mlsanity.com/paper/2409.04915/) (2024)
- [A Melting Pot of Evolution and Learning](https://mlsanity.com/paper/2306.04971/) (2023)
- [A Neural Architecture Search Method using Auxiliary Evaluation Metric based on ResNet Architecture](https://mlsanity.com/paper/2505.01313/) (2025)

## Abstract (arXiv)

Activation functions (AFs) play a pivotal role in the performance of neural networks. The Rectified Linear Unit (ReLU) is currently the most commonly used AF. Several replacements to ReLU have been suggested but improvements have proven inconsistent. Some AFs exhibit better performance for specific tasks, but it is hard to know a priori how to select the appropriate one(s). Studying both standard fully connected neural networks (FCNs) and convolutional neural networks (CNNs), we propose a novel, three-population, coevolutionary algorithm to evolve AFs, and compare it to four other methods, both evolutionary and non-evolutionary. Tested on four datasets -- MNIST, FashionMNIST, KMNIST, and USPS -- coevolution proves to be a performant algorithm for finding good AFs and AF architectures.