# Fast-Slow Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1705.08639 · HTML: https://mlsanity.com/paper/1705.08639/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2017.

## Built on by (most cited first, 12 of 12)

- [Efficient Transformers with Dynamic Token Pooling](https://mlsanity.com/paper/2211.09761/) (2022)
- [Clockwork Variational Autoencoders](https://mlsanity.com/paper/2102.09532/) (2021)
- [Recurrent neural networks: vanishing and exploding gradients are not the end of the story](https://mlsanity.com/paper/2405.21064/) (2024)
- [An Independently Learnable Hierarchical Model for Bilateral Control-Based Imitation Learning Applications](https://mlsanity.com/paper/2203.08619/) (2022)
- [Variational Predictive Routing with Nested Subjective Timescales](https://mlsanity.com/paper/2110.11236/) (2021)
- [Long-horizon video prediction using a dynamic latent hierarchy](https://mlsanity.com/paper/2212.14376/) (2022)
- [FLOWGEN: Fast and slow graph generation](https://mlsanity.com/paper/2207.07656/) (2022)
- [Exploration of Fast-Slow Latent Recurrence for Train-Short, Test-Long Generalization](https://mlsanity.com/paper/2604.01577/) (2026)
- [SUNTA: Hierarchical Video Prediction with Surprise-based Chunking](https://mlsanity.com/paper/2607.02087/) (2026)
- [A Quantitative Review on Language Model Efficiency Research](https://mlsanity.com/paper/2306.01768/) (2023)
- [Contextually Guided Transformers via Low-Rank Adaptation](https://mlsanity.com/paper/2506.05672/) (2025)
- [EVAF: A Test-Retest Protocol for Selective Parametric Consolidation](https://mlsanity.com/paper/2606.29916/) (2026)

## Abstract (arXiv)

Processing sequential data of variable length is a major challenge in a wide range of applications, such as speech recognition, language modeling, generative image modeling and machine translation. Here, we address this challenge by proposing a novel recurrent neural network (RNN) architecture, the Fast-Slow RNN (FS-RNN). The FS-RNN incorporates the strengths of both multiscale RNNs and deep transition RNNs as it processes sequential data on different timescales and learns complex transition functions from one time step to the next. We evaluate the FS-RNN on two character level language modeling data sets, Penn Treebank and Hutter Prize Wikipedia, where we improve state of the art results to 1.19 and 1.25 bits-per-character (BPC), respectively. In addition, an ensemble of two FS-RNNs achieves 1.20 BPC on Hutter Prize Wikipedia outperforming the best known compression algorithm with respect to the BPC measure. We also present an empirical investigation of the learning and network dynamics of the FS-RNN, which explains the improved performance compared to other RNN architectures. Our approach is general as any kind of RNN cell is a possible building block for the FS-RNN architecture, and thus can be flexibly applied to different tasks.