# Weight-Parameterization in Continuous Time Deep Neural Networks for Surrogate Modeling

arXiv: https://arxiv.org/abs/2507.22045 · HTML: https://mlsanity.com/paper/2507.22045/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 29 Jul 2025.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control](https://mlsanity.com/paper/1909.12077/) (2019)
- [Dissecting Neural ODEs](https://mlsanity.com/paper/2002.08071/) (2020)
- [Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks](https://mlsanity.com/paper/1709.06079/) (2017)
- [ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs](https://mlsanity.com/paper/1902.10298/) (2019)
- [On orthogonality and learning recurrent networks with long term dependencies](https://mlsanity.com/paper/1702.00071/) (2017)
- [Deep Learning via Dynamical Systems: An Approximation Perspective](https://mlsanity.com/paper/1912.10382/) (2019)
- [Discretize-Optimize vs. Optimize-Discretize for Time-Series Regression and Continuous Normalizing Flows](https://mlsanity.com/paper/2005.13420/) (2020)
- [Deep learning as optimal control problems: models and numerical methods](https://mlsanity.com/paper/1904.05657/) (2019)
- [Train Like a (Var)Pro: Efficient Training of Neural Networks with Variable Projection](https://mlsanity.com/paper/2007.13171/) (2020)
- [Predicting Change, Not States: An Alternate Framework for Neural PDE Surrogates](https://mlsanity.com/paper/2412.13074/) (2024)
- [Time Dependence in Non-Autonomous Neural ODEs](https://mlsanity.com/paper/2005.01906/) (2020)
- [Differential Equations for Continuous-Time Deep Learning](https://mlsanity.com/paper/2401.03965/) (2024)
- [Spline parameterization of neural network controls for deep learning](https://mlsanity.com/paper/2103.00301/) (2021)
- [NeuralPDR: Neural Differential Equations as surrogate models for Photodissociation Regions](https://mlsanity.com/paper/2506.14270/) (2025)

## Abstract (arXiv)

Continuous-time deep learning models, such as neural ordinary differential equations (ODEs), offer a promising framework for surrogate modeling of complex physical systems. A central challenge in training these models lies in learning expressive yet stable time-varying weights, particularly under computational constraints. This work investigates weight parameterization strategies that constrain the temporal evolution of weights to a low-dimensional subspace spanned by polynomial basis functions. We evaluate both monomial and Legendre polynomial bases within neural ODE and residual network (ResNet) architectures under discretize-then-optimize and optimize-then-discretize training paradigms. Experimental results across three high-dimensional benchmark problems show that Legendre parameterizations yield more stable training dynamics, reduce computational cost, and achieve accuracy comparable to or better than both monomial parameterizations and unconstrained weight models. These findings elucidate the role of basis choice in time-dependent weight parameterization and demonstrate that using orthogonal polynomial bases offers a favorable tradeoff between model expressivity and training efficiency.