# Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1708.06834 · HTML: https://mlsanity.com/paper/1708.06834/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Aug 2017.

## Built on by (most cited first, 30 of 49)

- [Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs](https://mlsanity.com/paper/2310.01801/) (2023)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [PonderNet: Learning to Ponder](https://mlsanity.com/paper/2107.05407/) (2021)
- [Efficient Transformers with Dynamic Token Pooling](https://mlsanity.com/paper/2211.09761/) (2022)
- [Closed-form Continuous-time Neural Models](https://mlsanity.com/paper/2106.13898/) (2021)
- [FrameExit: Conditional Early Exiting for Efficient Video Recognition](https://mlsanity.com/paper/2104.13400/) (2021)
- [UnICORNN: A recurrent model for learning very long time dependencies](https://mlsanity.com/paper/2103.05487/) (2021)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute](https://mlsanity.com/paper/2102.12459/) (2021)
- [Modeling Multi-Label Action Dependencies for Temporal Action Localization](https://mlsanity.com/paper/2103.03027/) (2021)
- [Skip-Convolutions for Efficient Video Processing](https://mlsanity.com/paper/2104.11487/) (2021)
- [ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queries](https://mlsanity.com/paper/2603.12942/) (2026)
- [Variable-rate discrete representation learning](https://mlsanity.com/paper/2103.06089/) (2021)
- [A Survey on Dynamic Neural Networks for Natural Language Processing](https://mlsanity.com/paper/2202.07101/) (2022)
- [Transkimmer: Transformer Learns to Layer-wise Skim](https://mlsanity.com/paper/2205.07324/) (2022)
- [Sparsely Changing Latent States for Prediction and Planning in Partially Observable Domains](https://mlsanity.com/paper/2110.15949/) (2021)
- [Block-Skim: Efficient Question Answering for Transformer](https://mlsanity.com/paper/2112.08560/) (2021)
- [Efficient Deep Visual and Inertial Odometry with Adaptive Visual Modality Selection](https://mlsanity.com/paper/2205.06187/) (2022)
- [Task-Customized Self-Supervised Pre-training with Scalable Dynamic Routing](https://mlsanity.com/paper/2205.13267/) (2022)
- [Learning Asynchronous and Sparse Human-Object Interaction in Videos](https://mlsanity.com/paper/2103.02758/) (2021)
- [TSFool: Crafting Highly-Imperceptible Adversarial Time Series through Multi-Objective Attack](https://mlsanity.com/paper/2209.06388/) (2022)
- [Learning Robust Dynamics through Variational Sparse Gating](https://mlsanity.com/paper/2210.11698/) (2022)
- [Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices](https://mlsanity.com/paper/2102.07071/) (2021)
- [Gradient Forward-Propagation for Large-Scale Temporal Video Modelling](https://mlsanity.com/paper/2106.08318/) (2021)
- [Event Neural Networks](https://mlsanity.com/paper/2112.00891/) (2021)
- [Sequence Learning Using Equilibrium Propagation](https://mlsanity.com/paper/2209.09626/) (2022)
- [Gated Recurrent Neural Networks with Weighted Time-Delay Feedback](https://mlsanity.com/paper/2212.00228/) (2022)
- [Dynamic Gated Recurrent Neural Network for Compute-efficient Speech Enhancement](https://mlsanity.com/paper/2408.12425/) (2024)
- [iLoRE: Dynamic Graph Representation with Instant Long-term Modeling and Re-occurrence Preservation](https://mlsanity.com/paper/2309.02012/) (2023)
- [Triangular Dropout: Variable Network Width without Retraining](https://mlsanity.com/paper/2205.01235/) (2022)

## Abstract (arXiv)

Recurrent Neural Networks (RNNs) continue to show outstanding performance in sequence modeling tasks. However, training RNNs on long sequences often face challenges like slow inference, vanishing gradients and difficulty in capturing long term dependencies. In backpropagation through time settings, these issues are tightly coupled with the large, sequential computational graph resulting from unfolding the RNN in time. We introduce the Skip RNN model which extends existing RNN models by learning to skip state updates and shortens the effective size of the computational graph. This model can also be encouraged to perform fewer state updates through a budget constraint. We evaluate the proposed model on various tasks and show how it can reduce the number of required RNN updates while preserving, and sometimes even improving, the performance of the baseline RNN models. Source code is publicly available at https://imatge-upc.github.io/skiprnn-2017-telecombcn/ .