mlsanityData to 7 Oct 2026

SMR: State Memory Replay for Long Sequence Modeling

arXiv 2405.17534 ↗First posted 27 May 2024cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Despite the promising performance of state space models (SSMs) in long sequence modeling, limitations still exist. Advanced SSMs like S5 and S6 (Mamba) in addressing non-uniform sampling, their recursive structures impede efficient SSM computation via convolution. To overcome compatibility limitations in parallel convolutional computation, this paper proposes a novel non-recursive non-uniform sample processing strategy. Theoretical analysis of SSMs through the lens of Event-Triggered Control (ETC) theory reveals the Non-Stable State (NSS) problem, where deviations from sampling point requirements lead to error transmission and accumulation, causing the divergence of the SSM's hidden state. Our analysis further reveals that adjustments of input sequences with early memories can mitigate the NSS problem, achieving Sampling Step Adaptation (SSA). Building on this insight, we introduce a simple yet effective plug-and-play mechanism, State Memory Replay (SMR), which utilizes learnable memories to adjust the current state with multi-step information for generalization at sampling points different from those in the training data. This enables SSMs to stably model varying sampling points. Experiments on long-range modeling tasks in autoregressive language modeling and Long Range Arena demonstrate the general effectiveness of the SMR mechanism for a series of SSM models.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 24 of its 32 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

24 of 32 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  8. 2020Rethinking Attention with Performerscited by 1,584
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  10. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  11. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  12. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
12 more
  1. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  2. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 128
  3. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  4. 2021Modeling Irregular Time Series with Continuous Recurrent Unitscited by 101
  5. 2018Learning long-range spatial dependencies with horizontal gated-recurrent unitscited by 98
  6. 2023Toeplitz Neural Network for Sequence Modelingcited by 43
  7. 2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37
  8. 2024Interactive Continual Learning: Fast and Slow Thinkingcited by 23
  9. 2021PermuteFormer: Efficient Relative Position Encoding for Long Sequencescited by 14
  10. 2024Contrastive Augmented Graph2Graph Memory Interaction for Few Shot Continual Learningcited by 7
  11. 2024Investigating Deep Watermark Security: An Adversarial Transferability Perspectivecited by 3
  12. 2018State measurement error-to-state stability results based on approximate discrete-time modelscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 2.

6 papers.

  1. 2024Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationcited by 37
  2. 2024Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decodingcited by 12
  3. 2024Towards Building Specialized Generalist AI with System 1 and System 2 Fusioncited by 8
  4. 2025CodeSSM: Towards State Space Models for Code Understandingcited by 3
  5. 2026Analysis of Long Range Dependency Understanding in State Space Modelscited by 1
  6. 2026Towards Understanding What State Space Models Learn About Codecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025MS-SSM: A Multi-Scale State Space Model for Efficient Sequence Modeling
  2. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba
  3. 2024S7: Selective and Simplified State Space Layers for Sequence Modeling
  4. 2026MuonSSM: Orthogonalizing State Space Models for Sequence Modeling
  5. 2024Mathematical Formalism for Memory Compression in Selective State Space Models
  6. 2025Compute-in-Memory Implementation of State Space Models for Event Sequence Processing
  7. 2026DSSMs: State Space Models with Explicit Memory via Delay Differential Equations
  8. 2026Mamba-3: Improved Sequence Modeling using State Space Principles