# Neuronal diversity can improve machine learning for physics and beyond

arXiv: https://arxiv.org/abs/2204.04348 · HTML: https://mlsanity.com/paper/2204.04348/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 75 references that are in the map.
- First posted on arXiv 9 Apr 2022.

## Builds on (most cited first, 25 of 25)

- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [Entropy-SGD: Biasing Gradient Descent Into Wide Valleys](https://mlsanity.com/paper/1611.01838/) (2016)
- [Stochastic Gradient Descent as Approximate Bayesian Inference](https://mlsanity.com/paper/1704.04289/) (2017)
- [An Investigation into Neural Net Optimization via Hessian Eigenvalue Density](https://mlsanity.com/paper/1901.10159/) (2019)
- [PyHessian: Neural Networks Through the Lens of the Hessian](https://mlsanity.com/paper/1912.07145/) (2019)
- [Equinox: neural networks in JAX via callable PyTrees and filtered transformations](https://mlsanity.com/paper/2111.00254/) (2021)
- [Hamiltonian Generative Networks](https://mlsanity.com/paper/1909.13789/) (2019)
- [Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances](https://mlsanity.com/paper/2105.12221/) (2021)
- [A survey on modern trainable activation functions](https://mlsanity.com/paper/2005.00817/) (2020)
- [Understanding Short-Horizon Bias in Stochastic Meta-Optimization](https://mlsanity.com/paper/1803.02021/) (2018)
- [Learning Activation Functions to Improve Deep Neural Networks](https://mlsanity.com/paper/1412.6830/) (2014)
- [A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularization](https://mlsanity.com/paper/2012.03801/) (2020)
- [Diversity Networks: Neural Network Compression Using Determinantal Point Processes](https://mlsanity.com/paper/1511.05077/) (2015)
- [Scaling Down Deep Learning with MNIST-1D](https://mlsanity.com/paper/2011.14439/) (2020)
- [Adaptable Hamiltonian neural networks](https://mlsanity.com/paper/2102.13235/) (2021)
- [Learning Combinations of Activation Functions](https://mlsanity.com/paper/1801.09403/) (2018)
- [Forecasting Hamiltonian dynamics without canonical coordinates](https://mlsanity.com/paper/2010.15201/) (2020)
- [On the Robustness of Convolutional Neural Networks to Internal Architecture and Weight Perturbations](https://mlsanity.com/paper/1703.08245/) (2017)
- [Toward an AI Physicist for Unsupervised Learning](https://mlsanity.com/paper/1810.10525/) (2018)
- [Stochastic Gradient Descent in Continuous Time: A Central Limit Theorem](https://mlsanity.com/paper/1710.04273/) (2017)
- [Frequently Asked Questions for: The Atoms of Neural Computation](https://mlsanity.com/paper/1410.8826/) (2014)
- [Mastering high-dimensional dynamics with Hamiltonian neural networks](https://mlsanity.com/paper/2008.04214/) (2020)

## Abstract (arXiv)

Diversity conveys advantages in nature, yet homogeneous neurons typically comprise the layers of artificial neural networks. Here we construct neural networks from neurons that learn their own activation functions, quickly diversify, and subsequently outperform their homogeneous counterparts on image classification and nonlinear regression tasks. Sub-networks instantiate the neurons, which meta-learn especially efficient sets of nonlinear responses. Examples include conventional neural networks classifying digits and forecasting a van der Pol oscillator and physics-informed Hamiltonian neural networks learning H'enon-Heiles stellar orbits and the swing of a video recorded pendulum clock. Such textitlearned diversity provides examples of dynamical systems selecting diversity over uniformity and elucidates the role of diversity in natural and artificial systems.