# Can recurrent neural networks warp time?

arXiv: https://arxiv.org/abs/1804.11188 · HTML: https://mlsanity.com/paper/1804.11188/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Mar 2018.

## Built on by (most cited first, 30 of 55)

- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [Hierarchically Gated Recurrent Neural Network for Sequence Modeling](https://mlsanity.com/paper/2311.04823/) (2023)
- [Theoretical Foundations of Deep Selective State-Space Models](https://mlsanity.com/paper/2402.19047/) (2024)
- [A Survey on Visual Mamba](https://mlsanity.com/paper/2404.15956/) (2024)
- [Long Expressive Memory for Sequence Modeling](https://mlsanity.com/paper/2110.04744/) (2021)
- [Mamba-R: Vision Mamba ALSO Needs Registers](https://mlsanity.com/paper/2405.14858/) (2024)
- [State-space models are accurate and efficient neural operators for dynamical systems](https://mlsanity.com/paper/2409.03231/) (2024)
- [Noether Networks: Meta-Learning Useful Conserved Quantities](https://mlsanity.com/paper/2112.03321/) (2021)
- [Recurrent neural networks: vanishing and exploding gradients are not the end of the story](https://mlsanity.com/paper/2405.21064/) (2024)
- [P-Mamba: Marrying Perona Malik Diffusion with Mamba for Efficient Pediatric Echocardiographic Left Ventricular Segmentation](https://mlsanity.com/paper/2402.08506/) (2024)
- [Traveling Waves Encode the Recent Past and Enhance Sequence Learning](https://mlsanity.com/paper/2309.08045/) (2023)
- [Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patching](https://mlsanity.com/paper/2405.20799/) (2024)
- [Evaluating the Robustness of Interpretability Methods through Explanation Invariance and Equivariance](https://mlsanity.com/paper/2304.06715/) (2023)
- [Learnable Commutative Monoids for Graph Neural Networks](https://mlsanity.com/paper/2212.08541/) (2022)
- [Adventurer: Optimizing Vision Mamba Architecture Designs for Efficiency](https://mlsanity.com/paper/2410.07599/) (2024)
- [Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models](https://mlsanity.com/paper/2406.04320/) (2024)
- [Slower is Better: Revisiting the Forgetting Mechanism in LSTM for Slower Information Decay](https://mlsanity.com/paper/2105.05944/) (2021)
- [Feature Programming for Multivariate Time Series Prediction](https://mlsanity.com/paper/2306.06252/) (2023)
- [Gated Recurrent Neural Networks with Weighted Time-Delay Feedback](https://mlsanity.com/paper/2212.00228/) (2022)
- [Time-Warping Invariant Quantum Recurrent Neural Networks via Quantum-Classical Adaptive Gating](https://mlsanity.com/paper/2301.08173/) (2023)
- [The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks](https://mlsanity.com/paper/2306.16922/) (2023)
- [Recurrent Neural Networks for Learning Long-term Temporal Dependencies with Reanalysis of Time Scale Representation](https://mlsanity.com/paper/2111.03282/) (2021)
- [Path sampling of recurrent neural networks by incorporating known physics](https://mlsanity.com/paper/2203.00597/) (2022)
- [Probable Object Location (POLo) Score Estimation for Efficient Object Goal Navigation](https://mlsanity.com/paper/2311.07992/) (2023)
- [FACTS: A Factored State-Space Framework For World Modelling](https://mlsanity.com/paper/2410.20922/) (2024)
- [Monocular visual simultaneous localization and mapping: (r)evolution from geometry to deep learning-based pipelines](https://mlsanity.com/paper/2503.02955/) (2025)
- [Online Stochastic Gradient Descent Learns Linear Dynamical Systems from A Single Trajectory](https://mlsanity.com/paper/2102.11822/) (2021)
- [Warming up recurrent neural networks to maximise reachable multistability greatly improves learning](https://mlsanity.com/paper/2106.01001/) (2021)
- [Inductive biases in deep learning models for weather prediction](https://mlsanity.com/paper/2304.04664/) (2023)

## Abstract (arXiv)

Successful recurrent models such as long short-term memories (LSTMs) and gated recurrent units (GRUs) use ad hoc gating mechanisms. Empirically these models have been found to improve the learning of medium to long term temporal dependencies and to help with vanishing gradient issues. We prove that learnable gates in a recurrent model formally provide quasi- invariance to general time transformations in the input data. We recover part of the LSTM architecture from a simple axiomatic approach. This result leads to a new way of initializing gate biases in LSTMs and GRUs. Ex- perimentally, this new chrono initialization is shown to greatly improve learning of long term dependencies, with minimal implementation effort.