# Initializing LSTM internal states via manifold learning

arXiv: https://arxiv.org/abs/2104.13101 · HTML: https://mlsanity.com/paper/2104.13101/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 59 references that are in the map.
- First posted on arXiv 27 Apr 2021.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [Searching for Activation Functions](https://mlsanity.com/paper/1710.05941/) (2017)
- [Data-Driven Forecasting of High-Dimensional Chaotic Systems with Long Short-Term Memory Networks](https://mlsanity.com/paper/1802.07486/) (2018)
- [Robust Learning with Jacobian Regularization](https://mlsanity.com/paper/1908.02729/) (2019)
- [Recent Advances in Recurrent Neural Networks](https://mlsanity.com/paper/1801.01078/) (2017)
- [Reversible Recurrent Neural Networks](https://mlsanity.com/paper/1810.10999/) (2018)
- [Controlling Recurrent Neural Networks by Conceptors](https://mlsanity.com/paper/1403.3369/) (2014)
- [Long-time predictive modeling of nonlinear dynamical systems using neural networks](https://mlsanity.com/paper/1805.12547/) (2018)

## Built on by (most cited first, 7 of 7)

- [A Framework for Machine Learning of Model Error in Dynamical Systems](https://mlsanity.com/paper/2107.06658/) (2021)
- [Tree-Wasserstein Distance for High Dimensional Data with a Latent Feature Hierarchy](https://mlsanity.com/paper/2410.21107/) (2024)
- [Some of the variables, some of the parameters, some of the times, with some physics known: Identification with partial information](https://mlsanity.com/paper/2304.14214/) (2023)
- [Learning Absorption Rates in Glucose-Insulin Dynamics from Meal Covariates](https://mlsanity.com/paper/2304.14300/) (2023)
- [Tailored Forecasting from Short Time Series via Meta-learning](https://mlsanity.com/paper/2501.16325/) (2025)
- [Next Generation Equation-Free Multiscale Modelling of Crowd Dynamics via Machine Learning](https://mlsanity.com/paper/2508.03926/) (2025)
- [Tipping Points of Evolving Epidemiological Networks: Machine Learning-Assisted, Data-Driven Effective Modeling](https://mlsanity.com/paper/2311.00797/) (2023)

## Abstract (arXiv)

We present an approach, based on learning an intrinsic data manifold, for the initialization of the internal state values of LSTM recurrent neural networks, ensuring consistency with the initial observed input data. Exploiting the generalized synchronization concept, we argue that the converged, "mature" internal states constitute a function on this learned manifold. The dimension of this manifold then dictates the length of observed input time series data required for consistent initialization. We illustrate our approach through a partially observed chemical model system, where initializing the internal LSTM states in this fashion yields visibly improved performance. Finally, we show that learning this data manifold enables the transformation of partially observed dynamics into fully observed ones, facilitating alternative identification paths for nonlinear dynamical systems.