# Warming up recurrent neural networks to maximise reachable multistability greatly improves learning

arXiv: https://arxiv.org/abs/2106.01001 · HTML: https://mlsanity.com/paper/2106.01001/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 39 references that are in the map.
- First posted on arXiv 2 Jun 2021.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [Hierarchical Multiscale Recurrent Neural Networks](https://mlsanity.com/paper/1609.01704/) (2016)
- [A Clockwork RNN](https://mlsanity.com/paper/1402.3511/) (2014)
- [Learning Longer-term Dependencies in RNNs with Auxiliary Losses](https://mlsanity.com/paper/1803.00144/) (2018)
- [Can recurrent neural networks warp time?](https://mlsanity.com/paper/1804.11188/) (2018)
- [Minimal Gated Unit for Recurrent Neural Networks](https://mlsanity.com/paper/1603.09420/) (2016)
- [Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics](https://mlsanity.com/paper/1906.10720/) (2019)
- [The unreasonable effectiveness of the forget gate](https://mlsanity.com/paper/1804.04849/) (2018)
- [Gated Orthogonal Recurrent Units: On Learning to Forget](https://mlsanity.com/paper/1706.02761/) (2017)
- [Interpreting recurrent neural networks behaviour via excitable network attractors](https://mlsanity.com/paper/1807.10478/) (2018)
- [Learning Longer Memory in Recurrent Neural Networks](https://mlsanity.com/paper/1412.7753/) (2014)
- [Recurrent Neural Network Training with Dark Knowledge Transfer](https://mlsanity.com/paper/1505.04630/) (2015)
- [A bio-inspired bistable recurrent cell allows for long-lasting memory](https://mlsanity.com/paper/2006.05252/) (2020)
- [Supervised level-wise pretraining for recurrent neural network initialization in multi-class classification](https://mlsanity.com/paper/1911.01071/) (2019)

## Built on by (most cited first, 3 of 3)

- [Parallelizable memory recurrent units](https://mlsanity.com/paper/2601.09495/) (2026)
- [Improving the Performance and Learning Stability of Parallelizable RNNs Designed for Ultra-Low Power Applications](https://mlsanity.com/paper/2605.11855/) (2026)
- [Train Short, Act Long: Multistability and the Horizon Independence of Recurrent Policies](https://mlsanity.com/paper/2605.12206/) (2026)

## Abstract (arXiv)

Training recurrent neural networks is known to be difficult when time dependencies become long. In this work, we show that most standard cells only have one stable equilibrium at initialisation, and that learning on tasks with long time dependencies generally occurs once the number of network stable equilibria increases; a property known as multistability. Multistability is often not easily attained by initially monostable networks, making learning of long time dependencies between inputs and outputs difficult. This insight leads to the design of a novel way to initialise any recurrent cell connectivity through a procedure called "warmup" to improve its capability to learn arbitrarily long time dependencies. This initialisation procedure is designed to maximise network reachable multistability, i.e., the number of equilibria within the network that can be reached through relevant input trajectories, in few gradient steps. We show on several information restitution, sequence classification, and reinforcement learning benchmarks that warming up greatly improves learning speed and performance, for multiple recurrent cells, but sometimes impedes precision. We therefore introduce a double-layer architecture initialised with a partial warmup that is shown to greatly improve learning of long time dependencies while maintaining high levels of precision. This approach provides a general framework for improving learning abilities of any recurrent cell when long time dependencies are present. We also show empirically that other initialisation and pretraining procedures from the literature implicitly foster reachable multistability of recurrent cells.