# Formalising the Use of the Activation Function in Neural Inference

arXiv: https://arxiv.org/abs/2102.04896 · HTML: https://mlsanity.com/paper/2102.04896/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 32 references that are in the map.
- First posted on arXiv 2 Feb 2021.

## Builds on (most cited first, 5 of 5)

- [On the Impact of the Activation Function on Deep Neural Networks Training](https://mlsanity.com/paper/1902.06853/) (2019)
- [Mean Field Residual Networks: On the Edge of Chaos](https://mlsanity.com/paper/1712.08969/) (2017)
- [Criticality in the brain: A synthesis of neurobiology, models and cognition](https://mlsanity.com/paper/1707.05952/) (2017)
- [Self-organized criticality in neural networks from activity-based rewiring](https://mlsanity.com/paper/2009.11781/) (2020)
- [Psychophysical identity and free energy](https://mlsanity.com/paper/2005.08935/) (2020)

## Built on by (most cited first, 1 of 1)

- [Nonlinearity Enhanced Adaptive Activation Functions](https://mlsanity.com/paper/2403.19896/) (2024)

## Abstract (arXiv)

We investigate how the activation function can be used to describe neural firing in an abstract way, and in turn, why it works well in artificial neural networks. We discuss how a spike in a biological neurone belongs to a particular universality class of phase transitions in statistical physics. We then show that the artificial neurone is, mathematically, a mean field model of biological neural membrane dynamics, which arises from modelling spiking as a phase transition. This allows us to treat selective neural firing in an abstract way, and formalise the role of the activation function in perceptron learning. The resultant statistical physical model allows us to recover the expressions for some known activation functions as various special cases. Along with deriving this model and specifying the analogous neural case, we analyse the phase transition to understand the physics of neural network learning. Together, it is shown that there is not only a biological meaning, but a physical justification, for the emergence and performance of typical activation functions; implications for neural learning and inference are also discussed.