mlsanityData to 7 Oct 2026

An Analytic Solution to Covariance Propagation in Neural Networks

arXiv 2403.16163 ↗First posted 24 Mar 2024cs.LG cs.AI stat.MLcited by 11

Not on the map. Why?

Abstract (arXiv)

Uncertainty quantification of neural networks is critical to measuring the reliability and robustness of deep learning systems. However, this often involves costly or inaccurate sampling methods and approximations. This paper presents a sample-free moment propagation technique that propagates mean vectors and covariance matrices across a network to accurately characterize the input-output distributions of neural networks. A key enabler of our technique is an analytic solution for the covariance of random variables passed through nonlinear activation functions, such as Heaviside, ReLU, and GELU. The wide applicability and merits of the proposed technique are shown in experiments analyzing the input-output distributions of trained neural networks and training Bayesian neural networks.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 17 of its 39 references that are in the map. First posted on arXiv 24 Mar 2024.

Builds on

17 of 39 references.

References · 17

  1. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  2. 2013Intriguing properties of neural networkscited by 5,114
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  8. 2014Striving for Simplicity: The All Convolutional Netcited by 983
  9. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  10. 2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
  11. 2019A General Framework for Uncertainty Estimation in Deep Learningcited by 119
  12. 2018Lightweight Probabilistic Deep Networkscited by 77
5 more
  1. 2013Tractable Inference for Complex Stochastic Processescited by 20
  2. 2017Semi-analytical approximations to statistical moments of sigmoid and softmax mappings of normal variablescited by 20
  3. 2019On the Sensitivity of Adversarial Robustness to Input Data Distributionscited by 20
  4. 2022How many perturbations break this model? Evaluating robustness beyond adversarial accuracycited by 9
  5. 2021On The Distribution of Penultimate Activations of Classification Networkscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 4.

11 papers.

  1. 2024Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learningcited by 2
  2. 2025Rethinking Approximate Gaussian Inference in Classificationcited by 2
  3. 2024A Decision-driven Methodology for Designing Uncertainty-aware AI Self-Assessmentcited by 1
  4. 2025A Guide to Failure in Machine Learning: Reliability and Robustness from Foundations to Practicecited by 0
  5. 2025ReLU Networks as Random Functions: Their Distribution in Probability Spacecited by 0
  6. 2025Optimized Weight Initialization on the Stiefel Manifold for Deep ReLU Neural Networkscited by 0
  7. 2025Kalman Bayesian Transformercited by 0
  8. 2026Uncertainty propagation through trained multi-layer perceptrons: Exact analytical resultscited by 0
  9. 2026Noise Stability of Transformer Modelscited by 0
  10. 2026Calibrated Sampling-Free Uncertainty Estimation in Bayesian Deep Learningcited by 0
  11. 2026Full-Covariance Smoothing of Bayesian Neural Networks for Online Adaptationcited by 0