# Neural Speed Reading via Skim-RNN

arXiv: https://arxiv.org/abs/1711.02085 · HTML: https://mlsanity.com/paper/1711.02085/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2017.

## Built on by (most cited first, 14 of 14)

- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs](https://mlsanity.com/paper/2310.01801/) (2023)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [Efficient Transformers with Dynamic Token Pooling](https://mlsanity.com/paper/2211.09761/) (2022)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [A Survey on Dynamic Neural Networks for Natural Language Processing](https://mlsanity.com/paper/2202.07101/) (2022)
- [Transkimmer: Transformer Learns to Layer-wise Skim](https://mlsanity.com/paper/2205.07324/) (2022)
- [Block-Skim: Efficient Question Answering for Transformer](https://mlsanity.com/paper/2112.08560/) (2021)
- [Efficient Deep Visual and Inertial Odometry with Adaptive Visual Modality Selection](https://mlsanity.com/paper/2205.06187/) (2022)
- [Flow: Per-Instance Personalized Federated Learning Through Dynamic Routing](https://mlsanity.com/paper/2211.15281/) (2022)
- [Balanced and Explainable Social Media Analysis for Public Health with Large Language Models](https://mlsanity.com/paper/2309.05951/) (2023)
- [Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices](https://mlsanity.com/paper/2102.07071/) (2021)
- [Part & Whole Extraction: Towards A Deep Understanding of Quantitative Facts for Percentages in Text](https://mlsanity.com/paper/2110.13505/) (2021)
- [SOI: Scaling Down Computational Complexity by Estimating Partial States of the Model](https://mlsanity.com/paper/2410.03813/) (2024)

## Abstract (arXiv)

Inspired by the principles of speed reading, we introduce Skim-RNN, a recurrent neural network (RNN) that dynamically decides to update only a small fraction of the hidden state for relatively unimportant input tokens. Skim-RNN gives computational advantage over an RNN that always updates the entire hidden state. Skim-RNN uses the same input and output interfaces as a standard RNN and can be easily used instead of RNNs in existing models. In our experiments, we show that Skim-RNN can achieve significantly reduced computational cost without losing accuracy compared to standard RNNs across five different natural language tasks. In addition, we demonstrate that the trade-off between accuracy and speed of Skim-RNN can be dynamically controlled during inference time in a stable manner. Our analysis also shows that Skim-RNN running on a single CPU offers lower latency compared to standard RNNs on GPUs.