Systematic construction of continuous-time neural networks for linear dynamical systems
Not on the map. Why?
Abstract (arXiv)
Discovering a suitable neural network architecture for modeling complex dynamical systems poses a formidable challenge, often involving extensive trial and error and navigation through a high-dimensional hyper-parameter space. In this paper, we discuss a systematic approach to constructing neural architectures for modeling a subclass of dynamical systems, namely, Linear Time-Invariant (LTI) systems. We use a variant of continuous-time neural networks in which the output of each neuron evolves continuously as a solution of a first-order or second-order Ordinary Differential Equation (ODE). Instead of deriving the network architecture and parameters from data, we propose a gradient-free algorithm to compute sparse architecture and network parameters directly from the given LTI system, leveraging its properties. We bring forth a novel neural architecture paradigm featuring horizontal hidden layers and provide insights into why employing conventional neural architectures with vertical hidden layers may not be favorable. We also provide an upper bound on the numerical errors of our neural networks. Finally, we demonstrate the high accuracy of our constructed networks on three numerical examples.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 38 of its 82 references that are in the map. First posted on arXiv 24 Mar 2024.
Builds on
38 of 82 references.
References · 38
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2018Neural Ordinary Differential Equationscited by 3,779
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
- 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
- 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 943
- 2018Efficient Neural Architecture Search via Parameter Sharingcited by 837
- 2019Importance Estimation for Neural Network Pruningcited by 541
- 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
- 2017Progressive Neural Architecture Searchcited by 487
- 2020A Comprehensive Survey of Neural Architecture Search: Challenges and Solutionscited by 254
- 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
26 more
- 2018Auto-Keras: An Efficient Neural Architecture Search Systemcited by 171
- 2019Quantization Networkscited by 153
- 2015Structured Pruning of Deep Convolutional Neural Networkscited by 149
- 2017Linearly-Recurrent Autoencoder Networks for Learning Dynamicscited by 145
- 2019Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruningcited by 89
- 2019ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEscited by 84
- 2020Learning Long-Term Dependencies in Irregularly-Sampled Time Seriescited by 83
- 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
- 2017Exploring Sparsity in Recurrent Neural Networkscited by 75
- 2018A Survey on Methods and Theories of Quantized Neural Networkscited by 75
- 2019A Survey on Neural Architecture Searchcited by 71
- 2021On the difficulty of learning chaotic dynamics with RNNscited by 67
- 2021Closed-form Continuous-time Neural Modelscited by 65
- 2016Deep vs. shallow networks : An approximation theory perspectivecited by 63
- 2022Stabilized Neural Ordinary Differential Equations for Long-Time Forecasting of Dynamical Systemscited by 50
- 2020Methods for Pruning Deep Neural Networkscited by 49
- 2019Adaptive Deep Learning for High-Dimensional Hamilton-Jacobi-Bellman Equationscited by 40
- 2020Deep Neural Networks for Nonlinear Model Order Reduction of Unsteady Flowscited by 36
- 2021Implicit energy regularization of neural ordinary-differential-equation controlcited by 29
- 2018Data-driven Discovery of Closure Modelscited by 20
- 2021A Neural Network Approach for High-Dimensional Optimal Control Applied to Multi-Agent Path Findingcited by 18
- 2023Neural Oscillators are Universalcited by 16
- 2023AONN: An adjoint-oriented neural network method for all-at-once solutions of parametric optimal control problemscited by 13
- 2022Dynamical systems' based neural networkscited by 10
- 2020A Tailored Convolutional Neural Network for Nonlinear Manifold Learning of Computational Physics Data using Unstructured Spatial Discretizationscited by 6
- 2023Neural Control of Parametric Solutions for High-dimensional Evolution PDEscited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.