# Per-Node Activation Function Evolution in Indirectly Encoded Substrates: Solvability, Limits, and Emergent Diversity

arXiv: https://arxiv.org/abs/2610.00149 · HTML: https://mlsanity.com/paper/2610.00149/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 10 Sep 2026.

## Builds on (most cited first, 12 of 12)

- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Neural Networks Fail to Learn Periodic Functions and How to Fix It](https://mlsanity.com/paper/2006.08195/) (2020)
- [Evolving Deep Neural Networks](https://mlsanity.com/paper/1703.00548/) (2017)
- [Discovering Parametric Activation Functions](https://mlsanity.com/paper/2006.03179/) (2020)
- [Growing Cosine Unit: A Novel Oscillatory Activation Function That Can Speedup Training and Reduce Parameters in Convolutional Neural Networks](https://mlsanity.com/paper/2108.12943/) (2021)
- [Evolving Parsimonious Networks by Mixing Activation Functions](https://mlsanity.com/paper/1703.07122/) (2017)
- [Evolution of Activation Functions: An Empirical Investigation](https://mlsanity.com/paper/2105.14614/) (2021)
- [Tensor-Accelerated Eager Multi-Resolution Grids for Evolving Large-Scale Substrates](https://mlsanity.com/paper/2608.27612/) (2026)
- [Investigating Hyperparameter Optimization and Transferability for ES-HyperNEAT: A TPE Approach](https://mlsanity.com/paper/2609.00449/) (2026)

## Built on by (most cited first, 1 of 1)

- [Bio-Inspired Palette Evolution in Indirectly Encoded Substrates: Timescale Compatibility Shapes Activation Function Discovery](https://mlsanity.com/paper/2609.17067/) (2026)

## Abstract (arXiv)

Biological neurons achieve computational diversity through specialized types: tonic, bursting, adapting, and fast-spiking cells coexist within the same circuit. Artificial neural networks, by contrast, apply a single activation function uniformly to all nodes, which limits what they can represent. We show that this uniformity creates hard limits for evolutionary search: across sparse evolved substrates, monotonic functions fail to solve parity beyond its smallest instance, XOR, while a single oscillatory unit suffices at all tested scales. The gap is one of search and sparsity, not representation: monotonic networks can represent parity with a modest number of hidden units, and gradient descent recovers that solution. We evolve, to our knowledge for the first time in indirect encoding, per-node activation function assignments from an 18-function palette across more than 4,500 experimental runs spanning Boolean logic, regression, and spatial classification. Testing each of the 18 functions individually on Parity-4 reveals a three-tier solvability structure: oscillatory functions achieve 100%, intermediate functions 6.7-80%, and all 9 monotonic functions 0%. This divide is not universal. Recurrence collapses it, and gradient descent inverts it entirely, showing that the barrier is specific to evolutionary search in sparse substrates. What activation functions a network can use, beyond its topology and weights, determines what evolutionary search can solve. Indirect encoding discovers heterogeneous per-node activation assignments unlikely to be chosen by hand.