# Rethinking Full Connectivity in Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1905.12340 · HTML: https://mlsanity.com/paper/1905.12340/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2019.

## Built on by (most cited first, 6 of 6)

- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Adaptive Slimming for Scalable and Efficient Speech Enhancement](https://mlsanity.com/paper/2507.04879/) (2025)
- [Direct Advantage Estimation for Scalable and Sample-efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2606.20411/) (2026)
- [Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compression](https://mlsanity.com/paper/2106.08301/) (2021)
- [W4S4: WaLRUS Meets S4 for Long-Range Sequence Modeling](https://mlsanity.com/paper/2506.07920/) (2025)
- [TaskSense: Focusing on What Matters in World Models](https://mlsanity.com/paper/2608.06544/) (2026)

## Abstract (arXiv)

Recurrent neural networks (RNNs) are omnipresent in sequence modeling tasks. Practical models usually consist of several layers of hundreds or thousands of neurons which are fully connected. This places a heavy computational and memory burden on hardware, restricting adoption in practical low-cost and low-power devices. Compared to fully convolutional models, the costly sequential operation of RNNs severely hinders performance on parallel hardware. This paper challenges the convention of full connectivity in RNNs. We study structurally sparse RNNs, showing that they are well suited for acceleration on parallel hardware, with a greatly reduced cost of the recurrent operations as well as orders of magnitude less recurrent weights. Extensive experiments on challenging tasks ranging from language modeling and speech recognition to video action recognition reveal that structurally sparse RNNs achieve competitive performance as compared to fully-connected networks. This allows for using large sparse RNNs for a wide range of real-world tasks that previously were too costly with fully connected networks.