mlsanityData to 7 Oct 2026

Efficient Long Sequence Modeling via State Space Augmented Transformer

arXiv 2212.08136 ↗First posted 15 Dec 2022cs.CL cs.LGcited by 37

Not on the map. Why?

Abstract (arXiv)

Transformer models have achieved superior performance in various natural language processing tasks. However, the quadratic computational cost of the attention mechanism limits its practicality for long sequences. There are existing attention variants that improve the computational efficiency, but they have limited ability to effectively compute global information. In parallel to Transformer models, state space models (SSMs) are tailored for long sequences, but they are not flexible enough to capture complicated local information. We propose SPADE, short for underlineState sunderlinePace underlineAugmenteunderlineD TransformunderlineEr. Specifically, we augment a SSM into the bottom layer of SPADE, and we employ efficient local attention methods for the other layers. The SSM augments global information, which complements the lack of long-range dependency issue in local attention methods. Experimental results on the Long Range Arena benchmark and language modeling tasks demonstrate the effectiveness of the proposed method. To further demonstrate the scalability of SPADE, we pre-train large encoder-decoder models and present fine-tuning results on natural language understanding and natural language generation tasks.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 59 of its 78 references that are in the map. First posted on arXiv 15 Dec 2022.

Builds on

59 of 78 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2016Layer Normalizationcited by 5,579
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
38 more
  1. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  2. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  3. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  5. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  6. 2020Reformer: The Efficient Transformercited by 1,658
  7. 2020Rethinking Attention with Performerscited by 1,584
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  9. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  11. 2018Self-Attention with Relative Position Representationscited by 1,112
  12. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  13. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  14. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  15. 2018Neural Network Acceptability Judgmentscited by 952
  16. 2020On Layer Normalization in the Transformer Architecturecited by 906
  17. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  18. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  19. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  20. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  21. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  22. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  23. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  24. 2021A Survey of Transformerscited by 399
  25. 2021Random Feature Attentioncited by 350
  26. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  27. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  28. 2021Efficient Attentions for Long Document Summarizationcited by 342
  29. 2020Sparse Sinkhorn Attentioncited by 250
  30. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  31. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  32. 2018A Simple Method for Commonsense Reasoningcited by 223
  33. 2020Understanding the Difficulty of Training Transformerscited by 210
  34. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  35. 2022Transformer Quality in Linear Timecited by 189
  36. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  37. 2019Blockwise Self-Attention for Long Document Understandingcited by 158
  38. 2019Adaptive Attention Span in Transformerscited by 148

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 19; 2025: 5; 2026: 1.

37 papers.

Sort:

By area:

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  2. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  3. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  4. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 104
  5. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  6. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  7. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  8. 2024MambaLLIE: Implicit Retinex-Aware Low Light Enhancement with Global-then-Local State Spacecited by 48
  9. 2023A Survey on Long Text Modeling with Transformerscited by 47
  10. 2024Mastering Memory Tasks with World Modelscited by 38
  11. 2023Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priorscited by 33
  12. 2023Facing Off World Model Backbones: RNNs, Transformers, and S4cited by 31
25 more
  1. 2024Dimba: Transformer-Mamba Diffusion Modelscited by 26
  2. 2024Matten: Video Generation with Mamba-Attentioncited by 24
  3. 20232-D SSM: A General Spatial Layer for Visual Transformerscited by 18
  4. 2024Slot State Space Modelscited by 17
  5. 2023Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layercited by 13
  6. 2025Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routingcited by 13
  7. 2023Focus Your Attention (with Adaptive IIR Filters)cited by 12
  8. 2025Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generationcited by 10
  9. 2023DiffECG: A Versatile Probabilistic Diffusion Model for ECG Signals Synthesiscited by 9
  10. 2024A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contextscited by 9
  11. 2023Hieros: Hierarchical Imagination on Structured State Space Sequence World Modelscited by 8
  12. 2024LOCOST: State-Space Models for Long Document Abstractive Summarizationcited by 8
  13. 2024Encoding and Controlling Global Semantics for Long-form Video Question Answeringcited by 8
  14. 2024SMR: State Memory Replay for Long Sequence Modelingcited by 6
  15. 2024A Survey of Transformer Enabled Time Series Synthesiscited by 5
  16. 2024Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequencescited by 5
  17. 2024LongVQ: Long Sequence Modeling with Vector Quantization on Structured Memorycited by 4
  18. 2025SAMBA: Toward a Long-Context EEG Foundation Model via Spatial Embedding and Differential Mambacited by 4
  19. 2025TransXSSM: A Hybrid Transformer State Space Model with Unified Rotary Position Embeddingcited by 3
  20. 2026Interpreting and Steering State-Space Models via Activation Subspace Bottleneckscited by 3
  21. 2023Fovea Transformer: Efficient Long-Context Modeling with Structured Fine-to-Coarse Attentioncited by 2
  22. 2024Reparameterized Multi-Resolution Convolutions for Long Sequence Modellingcited by 2
  23. 2025Technologies on Effectiveness and Efficiency: A Survey of State Spaces Modelscited by 2
  24. 2023Attention Is Not All You Need Anymorecited by 1
  25. 2024Venturing into Uncharted Waters: The Navigation Compass from Transformer to Mambacited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Block-State Transformers
  2. 2024Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms
  3. 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving
  4. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Models
  5. 2025Sliding Window Attention Training for Efficient Large Language Models
  6. 2023A Quantitative Review on Language Model Efficiency Research
  7. 2023Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation
  8. 2026Long-Context Modeling via GSS-Transformer Hybrid Architecture with Learnable Mixing