# State Derivative Normalization for Continuous-Time Deep Neural Networks

arXiv: https://arxiv.org/abs/2401.02902 · HTML: https://mlsanity.com/paper/2401.02902/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 20 references that are in the map.
- First posted on arXiv 5 Jan 2024.

## Builds on (most cited first, 11 of 11)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [Nonlinear System Identification: A User-Oriented Roadmap](https://mlsanity.com/paper/1902.00683/) (2019)
- [Continuous-time system identification with neural networks: Model structures and fitting criteria](https://mlsanity.com/paper/2006.02915/) (2020)
- [A flexible state space model for learning nonlinear dynamical systems](https://mlsanity.com/paper/1603.05486/) (2016)
- [On the Initialization of Nonlinear LFR Model Identification with the Best Linear Approximation](https://mlsanity.com/paper/2004.05040/) (2020)
- [Improved Initialization of State-Space Artificial Neural Networks](https://mlsanity.com/paper/2103.14516/) (2021)
- [Continuous-time identification of dynamic state-space models by deep subspace encoding](https://mlsanity.com/paper/2204.09405/) (2022)
- [Efficient Multidimensional Regularization for Volterra Series Estimation](https://mlsanity.com/paper/1804.10026/) (2018)

## Abstract (arXiv)

The importance of proper data normalization for deep neural networks is well known. However, in continuous-time state-space model estimation, it has been observed that improper normalization of either the hidden state or hidden state derivative of the model estimate, or even of the time interval can lead to numerical and optimization challenges with deep learning based methods. This results in a reduced model quality. In this contribution, we show that these three normalization tasks are inherently coupled. Due to the existence of this coupling, we propose a solution to all three normalization challenges by introducing a normalization constant at the state derivative level. We show that the appropriate choice of the normalization constant is related to the dynamics of the to-be-identified system and we derive multiple methods of obtaining an effective normalization constant. We compare and discuss all the normalization strategies on a benchmark problem based on experimental data from a cascaded tanks system and compare our results with other methods of the identification literature.