mlsanityData to 14 Oct 2026

Long Range Language Modeling via Gated State Spaces

arXiv 2206.13947 ↗First posted 27 Jun 2022cs.LG cs.CLcited by 265

Not on the map. Why?

Cited by 139 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

State space models have shown to be effective at modeling long range dependencies, specially on sequence classification tasks. In this work we focus on autoregressive sequence modeling over English books, Github source code and ArXiv mathematics articles. Based on recent developments around the effectiveness of gated activation functions, we propose a new layer named Gated State Space (GSS) and show that it trains significantly faster than the diagonal version of S4 (i.e. DSS) on TPUs, is fairly competitive with several well-tuned Transformer-based baselines and exhibits zero-shot generalization to longer inputs while being straightforward to implement. Finally, we show that leveraging self-attention to model local dependencies improves the performance of GSS even further.

Key facts. Cited by 265 papers in the map (data through 14 Oct 2026). Builds on 44 of its 53 references that are in the map. First posted on arXiv 27 Jun 2022.

Builds on

44 of 53 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2020Longformer: The Long-Document Transformercited by 3,413
  11. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  12. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
32 more
  1. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  4. 2020GLU Variants Improve Transformercited by 1,804
  5. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  6. 2020Reformer: The Efficient Transformercited by 1,668
  7. 2020Rethinking Attention with Performerscited by 1,601
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  10. 2020Efficient Transformers: A Surveycited by 790
  11. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  12. 2016Language Modeling with Gated Convolutional Networkscited by 702
  13. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  14. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  15. 2020Hopfield Networks is All You Needcited by 475
  16. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  17. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  18. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
  19. 2021Random Feature Attentioncited by 354
  20. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  21. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  22. 2022Memorizing Transformerscited by 226
  23. 2022cosFormer: Rethinking Softmax in Attentioncited by 213
  24. 2022Transformer Quality in Linear Timecited by 191
  25. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 162
  26. 2019Blockwise Self-Attention for Long Document Understandingcited by 159
  27. 2020Fast Transformers with Clustered Attentioncited by 123
  28. 2022Block-Recurrent Transformerscited by 110
  29. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  30. 2020GMAT: Global Memory Augmentation for Transformerscited by 36
  31. 2020ETC: Encoding Long and Structured Inputs in Transformerscited by 34
  32. 2021When Attention Meets Fast Recurrence: Training Language Models with Reduced Computecited by 34

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 31; 2024: 142; 2025: 66; 2026: 19.

265 papers.

Sort:

By area:

  1. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
  2. 2024VMamba: Visual State Space Modelcited by 969
  3. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  4. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  5. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  6. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  7. 2024MambaIR: A Simple Baseline for Image Restoration with State-Space Modelcited by 352
  8. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 328
  9. 2024Titans: Learning to Memorize at Test Timecited by 328
  10. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
  11. 2024LocalMamba: Visual State Space Model with Windowed Selective Scancited by 236
  12. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
38 more
  1. 2023Efficient Large Language Models: A Surveycited by 186
  2. 2022Mega: Moving Average Equipped Gated Attentioncited by 179
  3. 2024U-KAN Makes Strong Backbone for Medical Image Segmentation and Generationcited by 179
  4. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 174
  5. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 167
  6. 2024Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spacescited by 134
  7. 2024Mamba-UNet: UNet-Like Pure Visual Mamba for Medical Image Segmentationcited by 133
  8. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 127
  9. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 122
  10. 2024Transformers are Multi-State RNNscited by 120
  11. 2024MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detectioncited by 113
  12. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 108
  13. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 105
  14. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 102
  15. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  16. 2024The Hidden Attention of Mamba Modelscited by 91
  17. 2024VmambaIR: Visual State Space Model for Image Restorationcited by 90
  18. 2024In-Context Language Learning: Architectures and Algorithmscited by 89
  19. 2024SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time seriescited by 82
  20. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  21. 2023Diffusion Models Without Attentioncited by 75
  22. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 74
  23. 2024Pan-Mamba: Effective pan-sharpening with State Space Modelcited by 73
  24. 2024Point Mamba: A Novel Point Cloud Backbone Based on State Space Model with Octree-Based Ordering Strategycited by 73
  25. 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
  26. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Modelscited by 67
  27. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 64
  28. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 64
  29. 2023Simple Hardware-Efficient Long Convolutions for Sequence Modelingcited by 62
  30. 2024MambaByte: Token-free Selective State Space Modelcited by 59
  31. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  32. 2024Longhorn: State Space Models are Amortized Online Learnerscited by 59
  33. 2024FreqMamba: Viewing Mamba from a Frequency Perspective for Image Derainingcited by 57
  34. 2024Gamba: Marry Gaussian Splatting with Mamba for single view 3D reconstructioncited by 56
  35. 2024Diffusion Models and Representation Learning: A Surveycited by 54
  36. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  37. 2022Pretraining Without Attentioncited by 51
  38. 2022Efficient Movie Scene Detection using State-Space Transformerscited by 51

50 most cited of 265 All 265 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LongSSM: On the Length Extension of State-space Models in Language Modelling
  2. 2022Efficient Long Sequence Modeling via State Space Augmented Transformer
  3. 2023State Spaces Aren't Enough: Machine Translation Needs Attention
  4. 2023State space models can express n-gram languages
  5. 2024On the Expressiveness and Length Generalization of Selective State-Space Models on Regular Languages
  6. 2023State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memory
  7. 2025Long-Short Alignment for Effective Long-Context Modeling in LLMs
  8. 2026On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models