# Deep networks for system identification: a Survey

arXiv: https://arxiv.org/abs/2301.12832 · HTML: https://mlsanity.com/paper/2301.12832/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 177 of its 390 references that are in the map.
- First posted on arXiv 30 Jan 2023.

## Builds on (most cited first, 30 of 177)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)

## Built on by (most cited first, 30 of 37)

- [An L-BFGS-B approach for linear and nonlinear system identification under ℓ1 and group-Lasso regularization](https://mlsanity.com/paper/2403.03827/) (2024)
- [On the adaptation of in-context learners for system identification](https://mlsanity.com/paper/2312.04083/) (2023)
- [Adapting to Change: A Comparison of Continual and Transfer Learning for Modeling Building Thermal Dynamics under Concept Drifts](https://mlsanity.com/paper/2508.21615/) (2025)
- [Deep Learning of Dynamic Systems using System Identification Toolbox(TM)](https://mlsanity.com/paper/2409.07642/) (2024)
- [Online Control-Informed Learning](https://mlsanity.com/paper/2410.03924/) (2024)
- [Combining Federated Learning and Control: A Survey](https://mlsanity.com/paper/2407.11069/) (2024)
- [Introducing Interval Neural Networks for Uncertainty-Aware System Identification](https://mlsanity.com/paper/2504.18845/) (2025)
- [State-Space Kolmogorov Arnold Networks for Interpretable Nonlinear System Identification](https://mlsanity.com/paper/2506.16392/) (2025)
- [Efficient identification of linear, parameter-varying, and nonlinear systems with noise models](https://mlsanity.com/paper/2504.11982/) (2025)
- [Nonconvex Linear System Identification with Minimal State Representation](https://mlsanity.com/paper/2504.18791/) (2025)
- [System Identification Using Kolmogorov-Arnold Networks: A Case Study on Buck Converters](https://mlsanity.com/paper/2506.10434/) (2025)
- [Interpretable neural network system identification method for two families of second-order systems based on characteristic curves](https://mlsanity.com/paper/2509.10632/) (2025)
- [Stable-by-Design Neural Network-Based LPV State-Space Models for System Identification](https://mlsanity.com/paper/2510.24757/) (2025)
- [SOLIS: Physics-Informed Learning of Interpretable Neural Surrogates for Nonlinear Systems](https://mlsanity.com/paper/2604.14879/) (2026)
- [Assumed Density Filtering and Smoothing with Neural Network Surrogate Models](https://mlsanity.com/paper/2511.09016/) (2025)
- [Tuning the burn-in phase in training recurrent neural networks improves their performance](https://mlsanity.com/paper/2602.10911/) (2026)
- [Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignment](https://mlsanity.com/paper/2602.14844/) (2026)
- [PH-KAN: Port-Hamiltonian Kolmogorov-Arnold Network](https://mlsanity.com/paper/2606.14708/) (2026)
- [Dealing with Collinearity in Large-Scale Linear System Identification Using Gaussian Regression](https://mlsanity.com/paper/2302.10959/) (2023)
- [Exploiting the capacity of deep networks only at training stage for nonlinear black-box system identification](https://mlsanity.com/paper/2312.15969/) (2023)
- [Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient](https://mlsanity.com/paper/2409.15393/) (2024)
- [Data-Based Efficient Off-Policy Stabilizing Optimal Control Algorithms for Discrete-Time Linear Systems via Damping Coefficients](https://mlsanity.com/paper/2412.20845/) (2024)
- [A State Alignment-Centric Approach to Federated System Identification: The FedAlign Framework](https://mlsanity.com/paper/2503.12137/) (2025)
- [S-Crescendo: A Nested Transformer Weaving Framework for Scalable Nonlinear System in S-Domain Representation](https://mlsanity.com/paper/2505.11843/) (2025)
- [Modeling and Topology Estimation of Low Rank Dynamical Networks](https://mlsanity.com/paper/2511.06674/) (2025)
- [Comparison of neural network training strategies for the simulation of dynamical systems](https://mlsanity.com/paper/2512.03851/) (2025)
- [xFODE+: Explainable Type-2 Fuzzy Additive ODEs for Uncertainty Quantification](https://mlsanity.com/paper/2604.14880/) (2026)
- [xFODE: An Explainable Fuzzy Additive ODE Framework for System Identification](https://mlsanity.com/paper/2604.14883/) (2026)
- [Target Parameterization in Diffusion Models for Nonlinear Spatiotemporal System Identification](https://mlsanity.com/paper/2604.17566/) (2026)
- [Dissipative Latent Residual Physics-Informed Neural Networks for Modeling and Identification of Electromechanical Systems](https://mlsanity.com/paper/2604.18277/) (2026)

## Abstract (arXiv)

Deep learning is a topic of considerable current interest. The availability of massive data collections and powerful software resources has led to an impressive amount of results in many application areas that reveal essential but hidden properties of the observations. System identification learns mathematical descriptions of dynamic systems from input-output data and can thus benefit from the advances of deep neural networks to enrich the possible range of models to choose from. For this reason, we provide a survey of deep learning from a system identification perspective. We cover a wide spectrum of topics to enable researchers to understand the methods, providing rigorous practical and theoretical insights into the benefits and challenges of using them. The main aim of the identified model is to predict new data from previous observations. This can be achieved with different deep learning based modelling techniques and we discuss architectures commonly adopted in the literature, like feedforward, convolutional, and recurrent networks. Their parameters have to be estimated from past data trying to optimize the prediction performance. For this purpose, we discuss a specific set of first-order optimization tools that is emerged as efficient. The survey then draws connections to the well-studied area of kernel-based methods. They control the data fit by regularization terms that penalize models not in line with prior assumptions. We illustrate how to cast them in deep architectures to obtain deep kernel-based methods. The success of deep learning also resulted in surprising empirical observations, like the counter-intuitive behaviour of models with many parameters. We discuss the role of overparameterized models, including their connection to kernels, as well as implicit regularization mechanisms which affect generalization, specifically the interesting phenomena of benign overfitting ...