mlsanityData to 7 Oct 2026

Simple Recurrent Units for Highly Parallelizable Recurrence

arXiv 1709.02755 ↗First posted 8 Sep 2017cs.CL cs.NEcited by 60

Not on the map. Why?

Abstract (arXiv)

Common recurrent neural architectures scale poorly due to the intrinsic difficulty in parallelizing their state computations. In this work, we propose the Simple Recurrent Unit (SRU), a light recurrent unit that balances model capacity and scalability. SRU is designed to provide expressive recurrence, enable highly parallelized implementation, and comes with careful initialization to facilitate training of deep models. We demonstrate the effectiveness of SRU on multiple NLP tasks. SRU achieves 5--9x speed-up over cuDNN-optimized LSTM on classification and question answering datasets, and delivers stronger results than LSTM and convolutional models. We also obtain an average of 0.7 BLEU improvement over the Transformer model on translation by incorporating SRU into the architecture.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Sep 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 16; 2022: 9; 2023: 13; 2024: 7; 2025: 11; 2026: 3.

60 papers.

Sort:

By area:

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  2. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  3. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  4. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  5. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  6. 2021Delving into Deep Imbalanced Regressioncited by 157
  7. 2022Block-Recurrent Transformerscited by 107
  8. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  9. 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 68
  10. 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
  11. 2023Efficient Neural Music Generationcited by 50
  12. 2023Parallelizing non-linear sequential models over the sequence lengthcited by 45
38 more
  1. 2024The Expressive Capacity of State Space Models: A Formal Language Perspectivecited by 45
  2. 2021ABC: Attention with Bounded-memory Controlcited by 36
  3. 2021When Attention Meets Fast Recurrence: Training Language Models with Reduced Computecited by 34
  4. 2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
  5. 2021How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wildcited by 30
  6. 2022Efficient recurrent architectures through activity sparsity and sparse back-propagation through timecited by 26
  7. 2021Real-world Video Deblurring: A Benchmark Dataset and An Efficient Recurrent Neural Networkcited by 25
  8. 2024MetaLA: Unified Optimal Linear Approximation to Softmax Attention Mapcited by 20
  9. 2023Exploring the Promise and Limits of Real-Time Recurrent Learningcited by 19
  10. 2021Heterogeneous Multi-task Learning with Expert Diversitycited by 18
  11. 2025StateSpaceDiffuser: Bringing Long Context to Diffusion World Modelscited by 16
  12. 2023Multi-Head State Space Model for Speech Recognitioncited by 14
  13. 2025Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrievalcited by 13
  14. 2025ParaRNN: Unlocking Parallel Training of Nonlinear RNNs for Large Language Modelscited by 13
  15. 2025Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformerscited by 12
  16. 2022A Call for Clarity in Beam Search: How It Works and When It Stopscited by 10
  17. 2024TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction for Efficient Speech Separationcited by 10
  18. 2021CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognitioncited by 8
  19. 2025Sequential-Parallel Duality in Prefix Scannable Modelscited by 8
  20. 2021On the quantization of recurrent neural networkscited by 7
  21. 2021SRU++: Pioneering Fast Recurrence with Attention for Speech Recognitioncited by 6
  22. 2023Hyperspectral Image Denoising via Spatial-Spectral Recurrent Transformercited by 6
  23. 2024Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyphcited by 6
  24. 2025Fast weight programming and linear transformers: from machine learning to neurobiologycited by 6
  25. 2021Doping: A technique for efficient compression of LSTM models using sparse structured additive matricescited by 5
  26. 2021Neural networks can understand compositional functions that humans do not, in the context of emergent communicationcited by 4
  27. 2021STRODE: Stochastic Boundary Ordinary Differential Equationcited by 4
  28. 2022Simple Recurrence Improves Masked Language Modelscited by 4
  29. 2021TGIF: Tree-Graph Integrated-Format Parser for Enhanced UD with Two-Stage Generic- to Individual-Language Finetuningcited by 3
  30. 2022Exploiting Expert Knowledge for Assigning Firms to Industries: A Novel Deep Learning Methodcited by 3
  31. 2025Joint Resource Management for Energy-efficient UAV-assisted SWIPT-MEC: A Deep Reinforcement Learning Approachcited by 3
  32. 2025Recurrence Meets Transformers for Universal Multimodal Retrievalcited by 3
  33. 2023Dual-Attention Neural Transducers for Efficient Wake Word Spotting in Speech Recognitioncited by 2
  34. 2023RRWKV: Capturing Long-range Dependencies in RWKVcited by 2
  35. 2023RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech Separationcited by 2
  36. 2022Boosting the interpretability of clinical risk scores with intervention predictionscited by 1
  37. 2023RNN-Guard: Certified Robustness Against Multi-frame Attacks for Recurrent Neural Networkscited by 1
  38. 2024Birdie: Advancing State Space Models with Reward-Driven Objectives and Curriculacited by 1

50 most cited of 60 All 60 (JSON)