# Sparse Bayesian Deep Learning for Dynamic System Identification

arXiv: https://arxiv.org/abs/2107.12910 · HTML: https://mlsanity.com/paper/2107.12910/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 62 references that are in the map.
- First posted on arXiv 27 Jul 2021.

## Builds on (most cited first, 17 of 17)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization](https://mlsanity.com/paper/1406.2572/) (2014)
- [Scalable Bayesian Optimization Using Deep Neural Networks](https://mlsanity.com/paper/1502.05700/) (2015)
- [Practical Gauss-Newton Optimisation for Deep Learning](https://mlsanity.com/paper/1706.03662/) (2017)
- [The Case for Bayesian Deep Learning](https://mlsanity.com/paper/2001.10995/) (2020)
- [BayesNAS: A Bayesian Approach for Neural Architecture Search](https://mlsanity.com/paper/1905.04919/) (2019)
- [A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Network](https://mlsanity.com/paper/2102.02410/) (2021)
- [Deep State Space Models for Nonlinear System Identification](https://mlsanity.com/paper/2003.14162/) (2020)
- [Continuous-time system identification with neural networks: Model structures and fitting criteria](https://mlsanity.com/paper/2006.02915/) (2020)
- [Nonlinear state-space identification using deep encoder networks](https://mlsanity.com/paper/2012.07697/) (2020)
- [A flexible state space model for learning nonlinear dynamical systems](https://mlsanity.com/paper/1603.05486/) (2016)
- [Data-driven Discovery of Cyber-Physical Systems](https://mlsanity.com/paper/1810.00697/) (2018)
- [Nonlinear system identification with regularized Tensor Network B-splines](https://mlsanity.com/paper/2003.07594/) (2020)
- [Toolbox for Discovering Dynamic System Relations via TAG Guided Genetic Programming](https://mlsanity.com/paper/2012.08834/) (2020)
- [Non-Autoregressive vs Autoregressive Neural Networks for System Identification](https://mlsanity.com/paper/2105.02027/) (2021)

## Built on by (most cited first, 9 of 9)

- [Deep networks for system identification: a Survey](https://mlsanity.com/paper/2301.12832/) (2023)
- [Continuous-time identification of dynamic state-space models by deep subspace encoding](https://mlsanity.com/paper/2204.09405/) (2022)
- [Sparse deep neural networks for modeling aluminum electrolysis dynamics](https://mlsanity.com/paper/2209.05832/) (2022)
- [Sparse neural networks with skip-connections for identification of aluminum electrolysis cell](https://mlsanity.com/paper/2301.00582/) (2023)
- [Neural State-Space Models: Empirical Evaluation of Uncertainty Quantification](https://mlsanity.com/paper/2304.06349/) (2023)
- [Dealing with Collinearity in Large-Scale Linear System Identification Using Gaussian Regression](https://mlsanity.com/paper/2302.10959/) (2023)
- [Exploiting the capacity of deep networks only at training stage for nonlinear black-box system identification](https://mlsanity.com/paper/2312.15969/) (2023)
- [Bayesian Modeling and Estimation of Linear Time-Varying Systems using Neural Networks and Gaussian Processes](https://mlsanity.com/paper/2507.12878/) (2025)
- [In-Context Learning for System Identification: Probabilistic Formulation, Long-Sequence Handling, and Benchmark Validation](https://mlsanity.com/paper/2610.13411/) (2026)

## Abstract (arXiv)

This paper proposes a sparse Bayesian treatment of deep neural networks (DNNs) for system identification. Although DNNs show impressive approximation ability in various fields, several challenges still exist for system identification problems. First, DNNs are known to be too complex that they can easily overfit the training data. Second, the selection of the input regressors for system identification is nontrivial. Third, uncertainty quantification of the model parameters and predictions are necessary. The proposed Bayesian approach offers a principled way to alleviate the above challenges by marginal likelihood/model evidence approximation and structured group sparsity-inducing priors construction. The identification algorithm is derived as an iterative regularised optimisation procedure that can be solved as efficiently as training typical DNNs. Remarkably, an efficient and recursive Hessian calculation method for each layer of DNNs is developed, turning the intractable training/optimisation process into a tractable one. Furthermore, a practical calculation approach based on the Monte-Carlo integration method is derived to quantify the uncertainty of the parameters and predictions. The effectiveness of the proposed Bayesian approach is demonstrated on several linear and nonlinear system identification benchmarks by achieving good and competitive simulation accuracy. The code to reproduce the experimental results is open-sourced and available online.