# Variable Computation in Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1611.06188 · HTML: https://mlsanity.com/paper/1611.06188/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2016.

## Built on by (most cited first, 10 of 10)

- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [Confident Adaptive Language Modeling](https://mlsanity.com/paper/2207.07061/) (2022)
- [Mixture-of-Depths: Dynamically allocating compute in transformer-based language models](https://mlsanity.com/paper/2404.02258/) (2024)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Attention Mechanism in Neural Networks: Where it Comes and Where it Goes](https://mlsanity.com/paper/2204.13154/) (2022)
- [A Survey on Dynamic Neural Networks for Natural Language Processing](https://mlsanity.com/paper/2202.07101/) (2022)
- [Triangular Dropout: Variable Network Width without Retraining](https://mlsanity.com/paper/2205.01235/) (2022)
- [Rule Based Learning with Dynamic (Graph) Neural Networks](https://mlsanity.com/paper/2406.09954/) (2024)
- [SOI: Scaling Down Computational Complexity by Estimating Partial States of the Model](https://mlsanity.com/paper/2410.03813/) (2024)
- [SkipSNN: Efficiently Classifying Spike Trains with Event-attention](https://mlsanity.com/paper/2411.05806/) (2024)

## Abstract (arXiv)

Recurrent neural networks (RNNs) have been used extensively and with increasing success to model various types of sequential data. Much of this progress has been achieved through devising recurrent units and architectures with the flexibility to capture complex statistics in the data, such as long range dependency or localized attention phenomena. However, while many sequential data (such as video, speech or language) can have highly variable information flow, most recurrent models still consume input features at a constant rate and perform a constant number of computations per time step, which can be detrimental to both speed and model capacity. In this paper, we explore a modification to existing recurrent units which allows them to learn to vary the amount of computation they perform at each step, without prior knowledge of the sequence's time structure. We show experimentally that not only do our models require fewer operations, they also lead to better performance overall on evaluation tasks.