GNet: A scalable and flexible Gaussian process network with nonparametric neurons

We build GNet, a scalable and flexible Gaussian process network with nonparametric activation functions. The key computational contribution is to make estimation of GP activations computationally scalable without forming covariance matrices explicitly. We provide theoretical studies comparing function classes with a fixed architecture, and derive predictive risk bounds for GNets. Using essentially the same architecture and optimization settings, GNet shows competitive performance across a diverse range of test problems, including predicting nonlinear functions, nonparametric regression of real-world data, and predicting one-body direct correlation functions with high-dimensional inputs in classical density functional theory. The strong performance of GNet suggests that flexible nonparametric neurons can substantially reduce the number of trainable parameters while remaining computationally scalable.