# Evolving Parsimonious Networks by Mixing Activation Functions

arXiv: https://arxiv.org/abs/1703.07122 · HTML: https://mlsanity.com/paper/1703.07122/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Mar 2017.

## Built on by (most cited first, 8 of 8)

- [Class Binarization to NeuroEvolution for Multiclass Classification](https://mlsanity.com/paper/2308.13876/) (2023)
- [Learning to Act through Evolution of Neural Diversity in Random Neural Networks](https://mlsanity.com/paper/2305.15945/) (2023)
- [Activation Function Optimization Scheme for Image Classification](https://mlsanity.com/paper/2409.04915/) (2024)
- [Per-Node Activation Function Evolution in Indirectly Encoded Substrates: Solvability, Limits, and Emergent Diversity](https://mlsanity.com/paper/2610.00149/) (2026)
- [Modular Grammatical Evolution for the Generation of Artificial Neural Networks](https://mlsanity.com/paper/2208.02787/) (2022)
- [Local Control Networks (LCNs): Optimizing Flexibility in Neural Network Data Pattern Capture](https://mlsanity.com/paper/2501.14000/) (2025)
- [Bio-Inspired Palette Evolution in Indirectly Encoded Substrates: Timescale Compatibility Shapes Activation Function Discovery](https://mlsanity.com/paper/2609.17067/) (2026)
- [Multi-Behavioral Evolved Substrates Through Neuromodulation and Activation Selection](https://mlsanity.com/paper/2610.00148/) (2026)

## Abstract (arXiv)

Neuroevolution methods evolve the weights of a neural network, and in some cases the topology, but little work has been done to analyze the effect of evolving the activation functions of individual nodes on network size, which is important when training networks with a small number of samples. In this work we extend the neuroevolution algorithm NEAT to evolve the activation function of neurons in addition to the topology and weights of the network. The size and performance of networks produced using NEAT with uniform activation in all nodes, or homogenous networks, is compared to networks which contain a mixture of activation functions, or heterogenous networks. For a number of regression and classification benchmarks it is shown that, (1) qualitatively different activation functions lead to different results in homogeneous networks, (2) the heterogeneous version of NEAT is able to select well performing activation functions, (3) producing heterogeneous networks that are significantly smaller than homogeneous networks.