# Activation Saturation and Floquet Spectrum Collapse in Neural ODEs

arXiv: https://arxiv.org/abs/2604.00543 · HTML: https://mlsanity.com/paper/2604.00543/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 32 references that are in the map.
- First posted on arXiv 1 Apr 2026.

## Builds on (most cited first, 11 of 11)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models](https://mlsanity.com/paper/1810.01367/) (2018)
- [Neural Controlled Differential Equations for Irregular Time Series](https://mlsanity.com/paper/2005.08926/) (2020)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [On the Number of Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1402.1869/) (2014)
- [How to train your neural ODE: the world of Jacobian and kinetic regularization](https://mlsanity.com/paper/2002.02798/) (2020)
- [Dissecting Neural ODEs](https://mlsanity.com/paper/2002.08071/) (2020)
- [Global Convergence in Neural ODEs: Impact of Activation Functions](https://mlsanity.com/paper/2509.22436/) (2025)
- [Comparing Physics-Informed and Neural ODE Approaches for Modeling Nonlinear Biological Systems: A Case Study Based on the Morris-Lecar Model](https://mlsanity.com/paper/2603.26921/) (2026)

## Built on by (most cited first, 1 of 1)

- [Long-Time Trajectory Approximation via SA-NODEs: Model Predictive and Floquet Strategies](https://mlsanity.com/paper/2608.10738/) (2026)

## Abstract (arXiv)

We prove that activation saturation imposes a structural dynamical limitation on autonomous Neural ODEs doth=fθ(h) with saturating activations (tanh, sigmoid, etc.): if q hidden layers of the MLP fθ satisfy |σ'|≤δ on a region~U, the input Jacobian is attenuated as normDfθ(x)≤ C(U) (for activations with supx|σ'(x)|≤ 1, e.g. tanh and sigmoid, this reduces to CWδq), forcing every Floquet (Lyapunov) exponen along any T-periodic orbit γsubset U into the interval [-C(U),;C(U)]. This is a collapse of the Floquet spectrum: as saturation deepens (δ→ 0), all exponents are driven to zero, limiting both strong contraction and chaotic sensitivity. The obstruction is structural -- it constrains the learned vector field at inference time, independent of training quality. As a secondary contribution, for activations with σ'>0, a saturation-weighted spectral factorisation yields a refined bound widetildeC(U)≤ C(U) whose improvement is amplified exponentially in~T at the flow level. All results are numerically illustrated on the Stuart--Landau oscillator; the bounds provide a theoretical explanation for the empirically observed failure of tanh-NODEs on the Morris--Lecar neuron model.