# Neural network with optimal neuron activation functions based on additive Gaussian process regression

arXiv: https://arxiv.org/abs/2301.05567 · HTML: https://mlsanity.com/paper/2301.05567/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 79 references that are in the map.
- First posted on arXiv 13 Jan 2023.

## Builds on (most cited first, 4 of 4)

- [Machine Learning Force Fields](https://mlsanity.com/paper/2010.07067/) (2020)
- [SpookyNet: Learning Force Fields with Electronic Degrees of Freedom and Nonlocal Effects](https://mlsanity.com/paper/2105.00304/) (2021)
- [On the optimization of hyperparameters in Gaussian process regression with the help of low-order high-dimensional model representation](https://mlsanity.com/paper/2112.01374/) (2021)
- [Random Sampling High Dimensional Model Representation Gaussian Process Regression (RS-HDMR-GPR) for representing multidimensional functions with machine-learned lower-dimensional terms allowing insight with a general method](https://mlsanity.com/paper/2012.02704/) (2020)

## Built on by (most cited first, 5 of 5)

- [Degeneration of kernel regression with Matern kernels into low-order polynomial regression in high dimension](https://mlsanity.com/paper/2311.10790/) (2023)
- [Orders-of-coupling representation with a single neural network with optimal neuron activation functions and without nonlinear parameter optimization](https://mlsanity.com/paper/2302.12013/) (2023)
- [Gaussian Process Regression -- Neural Network Hybrid with Optimized Redundant Coordinates](https://mlsanity.com/paper/2509.08457/) (2025)
- [Neural networks for neurocomputing circuits: a computational study of tolerance to noise and activation function non-uniformity when machine learning materials properties](https://mlsanity.com/paper/2510.17849/) (2025)
- [GNet: A scalable and flexible Gaussian process network with nonparametric neurons](https://mlsanity.com/paper/2607.10735/) (2026)

## Abstract (arXiv)

Feed-forward neural networks (NN) are a staple machine learning method widely used in many areas of science and technology. While even a single-hidden layer NN is a universal approximator, its expressive power is limited by the use of simple neuron activation functions (such as sigmoid functions) that are typically the same for all neurons. More flexible neuron activation functions would allow using fewer neurons and layers and thereby save computational cost and improve expressive power. We show that additive Gaussian process regression (GPR) can be used to construct optimal neuron activation functions that are individual to each neuron. An approach is also introduced that avoids non-linear fitting of neural network parameters. The resulting method combines the advantage of robustness of a linear regression with the higher expressive power of a NN. We demonstrate the approach by fitting the potential energy surfaces of the water molecule and formaldehyde. Without requiring any non-linear optimization, the additive GPR based approach outperforms a conventional NN in the high accuracy regime, where a conventional NN suffers more from overfitting.