mlsanityData to 7 Oct 2026

Parallelizing Linear Recurrent Neural Nets Over Sequence Length

arXiv 1709.04057 ↗First posted 12 Sep 2017cs.NE cs.AI cs.LGcited by 126

Not on the map. Why?

Cited by 25 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recurrent neural networks (RNNs) are widely used to model sequential data but their non-linear dependencies between sequence elements prevent parallelizing training over sequence length. We show the training of RNNs with only linear sequential dependencies can be parallelized over the sequence length using the parallel scan algorithm, leading to rapid training on long sequences even with small minibatch size. We develop a parallel linear recurrence CUDA kernel and show that it can be applied to immediately speed up training and inference of several state of the art RNN architectures by up to 9x. We abstract recent work on linear RNNs into a new framework of linear surrogate RNNs and develop a linear surrogate model for the long short-term memory unit, the GILR-LSTM, that utilizes parallel linear recurrence. We extend sequence learning to new extremely long sequence regimes that were previously out of reach by successfully training a GILR-LSTM on a synthetic sequence classification task with a one million timestep dependency.

Key facts. Cited by 126 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Sep 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 16; 2024: 39; 2025: 40; 2026: 29.

126 papers.

Sort:

By area:

  1. 2024VMamba: Visual State Space Modelcited by 955
  2. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  3. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  4. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  5. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  6. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  7. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  8. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  9. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  10. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  11. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  12. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
38 more
  1. 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 108
  2. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  3. 2026Kimi K3: Open Frontier Intelligencecited by 99
  4. 2024The Hidden Attention of Mamba Modelscited by 88
  5. 2024Theoretical Foundations of Deep Selective State-Space Modelscited by 73
  6. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
  7. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 69
  8. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  9. 2023Parallelizing non-linear sequential models over the sequence lengthcited by 45
  10. 2023Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependenciescited by 41
  11. 2024Mamba-R: Vision Mamba ALSO Needs Registerscited by 41
  12. 2024MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selectioncited by 40
  13. 2023State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memorycited by 38
  14. 2024Towards Scalable and Stable Parallelization of Nonlinear RNNscited by 38
  15. 2023Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvaluescited by 35
  16. 2024Oscillatory State-Space Modelscited by 35
  17. 2023StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterizationcited by 29
  18. 2021Parallelizing Legendre Memory Unit Trainingcited by 22
  19. 2024MetaLA: Unified Optimal Linear Approximation to Softmax Attention Mapcited by 20
  20. 2024State-Free Inference of State-Space Models: The Transfer Function Approachcited by 18
  21. 2024On the Resurgence of Recurrent Models for Long Sequences -- Survey and Research Opportunities in the Transformer Eracited by 17
  22. 2025Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Modelscited by 16
  23. 2025TempoPFN: Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecastingcited by 16
  24. 2024RankMamba: Benchmarking Mamba's Document Ranking Performance in the Era of Transformerscited by 15
  25. 2024On the Expressiveness and Length Generalization of Selective State-Space Models on Regular Languagescited by 15
  26. 2025Structured Sparse Transition Matrices to Enable State Tracking in State-Space Modelscited by 15
  27. 2023Transformer Working Memory Enables Regular Language Reasoning and Natural Language Length Extrapolationcited by 13
  28. 2023Advancing Regular Language Reasoning in Linear Recurrent Neural Networkscited by 13
  29. 2024Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulationcited by 11
  30. 2025Prototype-Driven Structure Synergy Network for Remote Sensing Images Segmentationcited by 11
  31. 2024Real-Time Recurrent Learning using Trace Units in Reinforcement Learningcited by 10
  32. 2025Parallelization of Non-linear State-Space Models: Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modelingcited by 10
  33. 2024Does Transformer Interpretability Transfer to RNNs?cited by 9
  34. 2024You Only Scan Once: Efficient Multi-dimension Sequential Modeling with LightNetcited by 9
  35. 2024Linear Attention Sequence Parallelismcited by 8
  36. 2024Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Modelscited by 8
  37. 2025Sequential-Parallel Duality in Prefix Scannable Modelscited by 8
  38. 2025SHAP Meets Tensor Networks: Provably Tractable Explanations with Parallelismcited by 8

50 most cited of 126 All 126 (JSON)