# Improving the performance of Stein variational inference through extreme sparsification of physically-constrained neural network models

arXiv: https://arxiv.org/abs/2407.00761 · HTML: https://mlsanity.com/paper/2407.00761/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 55 references that are in the map.
- First posted on arXiv 30 Jun 2024.

## Builds on (most cited first, 24 of 24)

- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm](https://mlsanity.com/paper/1608.04471/) (2016)
- [Input Convex Neural Networks](https://mlsanity.com/paper/1609.07152/) (2016)
- [Advances in Variational Inference](https://mlsanity.com/paper/1711.05597/) (2017)
- [A Conceptual Introduction to Hamiltonian Monte Carlo](https://mlsanity.com/paper/1701.02434/) (2017)
- [Stein Variational Gradient Descent as Gradient Flow](https://mlsanity.com/paper/1704.07520/) (2017)
- [Bayesian Compression for Deep Learning](https://mlsanity.com/paper/1705.08665/) (2017)
- [Bayesian Bits: Unifying Quantization and Pruning](https://mlsanity.com/paper/2005.07093/) (2020)
- [Polyconvex anisotropic hyperelasticity with neural networks](https://mlsanity.com/paper/2106.14623/) (2021)
- [Projected Stein Variational Gradient Descent](https://mlsanity.com/paper/2002.03469/) (2020)
- [Data-driven Tissue Mechanics with Polyconvex Neural Ordinary Differential Equations](https://mlsanity.com/paper/2110.03774/) (2021)
- [Projected Stein Variational Newton: A Fast and Scalable Bayesian Inference Method in High Dimensions](https://mlsanity.com/paper/1901.08659/) (2019)
- [Modular machine learning-based elastoplasticity: generalization in the context of limited data](https://mlsanity.com/paper/2210.08343/) (2022)
- [Extreme sparsification of physics-augmented neural networks for interpretable model discovery in mechanics](https://mlsanity.com/paper/2310.03652/) (2023)
- [Stein Variational Gradient Descent as Moment Matching](https://mlsanity.com/paper/1810.11693/) (2018)
- [Neural network-based multiscale modeling of finite strain magneto-elasticity with relaxed convexity criteria](https://mlsanity.com/paper/2308.16311/) (2023)
- [Equivariant geometric learning for digital rock physics: estimating formation factor and effective permeability tensors from Morse graph](https://mlsanity.com/paper/2104.05608/) (2021)
- [Stress representations for tensor basis neural networks: alternative formulations to Finger-Rivlin-Ericksen](https://mlsanity.com/paper/2308.11080/) (2023)
- [On the detrimental effect of invariances in the likelihood for variational inference](https://mlsanity.com/paper/2209.07157/) (2022)
- [A stochastic Stein Variational Newton method](https://mlsanity.com/paper/2204.09039/) (2022)
- [Bridging scales with Machine Learning: From first principles statistical mechanics to continuum phase field computations to study order disorder transitions in LixCoO2](https://mlsanity.com/paper/2302.08991/) (2023)
- [Accurate Data-Driven Surrogates of Dynamical Systems for Forward Propagation of Uncertainty](https://mlsanity.com/paper/2310.10831/) (2023)

## Built on by (most cited first, 4 of 4)

- [Uncertainty quantification of neural network models of evolving processes via Langevin sampling](https://mlsanity.com/paper/2504.14854/) (2025)
- [Fully data-driven inverse hyperelasticity with hyper-network neural ODE fields](https://mlsanity.com/paper/2506.08146/) (2025)
- [CANN-EUCLID: unsupervised constitutive artificial neural network model discovery from full-field data](https://mlsanity.com/paper/2606.14565/) (2026)
- [Interval and fuzzy physics-augmented neural networks (iPANN and fPANN) for uncertainty quantification and propagation in constitutive modeling](https://mlsanity.com/paper/2607.20339/) (2026)

## Abstract (arXiv)

Most scientific machine learning (SciML) applications of neural networks involve hundreds to thousands of parameters, and hence, uncertainty quantification for such models is plagued by the curse of dimensionality. Using physical applications, we show that L0 sparsification prior to Stein variational gradient descent (L0+SVGD) is a more robust and efficient means of uncertainty quantification, in terms of computational cost and performance than the direct application of SGVD or projected SGVD methods. Specifically, L0+SVGD demonstrates superior resilience to noise, the ability to perform well in extrapolated regions, and a faster convergence rate to an optimal solution.