# Exact Gaussian Moment Matching for Residual Networks: a Second-Order Method

arXiv: https://arxiv.org/abs/2601.22307 · HTML: https://mlsanity.com/paper/2601.22307/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jan 2026.

## Built on by (most cited first, 1 of 1)

- [Uncertainty propagation in auto-regressive random neural network models](https://mlsanity.com/paper/2608.20483/) (2026)

## Abstract (arXiv)

We study the problem of propagating the mean and covariance of a general multivariate Gaussian distribution through a deep (residual) neural network using layer-by-layer moment matching. We close a longstanding gap by deriving exact moment matching for the probit, GeLU, ReLU (as a limit of GeLU), Heaviside (as a limit of probit), and sine activation functions; for both feedforward and generalized residual layers. On random networks, we find orders-of-magnitude improvements in the KL divergence error metric, up to a millionfold, over popular alternatives. On a variational Bayes neural network, we show that our method attains hundredfold improvements in KL divergence from Monte Carlo ground truth over a state-of-the-art deterministic inference method. We also give a smooth-distance error bound showing that, under regularity assumptions, moment matching removes the leading low-variance errors and propagates higher-order local accuracy through the layers of a network.