mlsanityData to 14 Oct 2026

Emergence of the Primacy Effect in Structured State-Space Models

arXiv 2502.13729 ↗First posted 19 Feb 2025cs.LG cs.NE q-bio.NCcited by 4

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Structured state-space models (SSMs) have been developed to offer more persistent memory retention than traditional recurrent neural networks, while maintaining real-time inference capabilities and addressing the time-complexity limitations of Transformers. Despite this intended persistence, the memory mechanism of canonical SSMs is theoretically designed to decay monotonically over time, meaning that more recent inputs are expected to be retained more accurately than earlier ones. Contrary to this theoretical expectation, however, the present study reveals a counterintuitive finding: when trained and evaluated on a synthetic, statistically balanced memorization task, SSMs predominantly preserve the *initially* presented data in memory. This pattern of memory bias, known as the *primacy effect* in psychology, presents a non-trivial challenge to the current theoretical understanding of SSMs and opens new avenues for future research.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 30 of its 61 references that are in the map. First posted on arXiv 19 Feb 2025.

Builds on

30 of 61 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  3. 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections24 Jun
  4. 2022Diagonal State Spaces are as Effective as Structured State Spaces27 Mar
  5. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  8. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,436
  9. 2022In-context Learning and Induction Headscited by 964
  10. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 608
  11. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  12. 2013Generating Sequences With Recurrent Neural Networkscited by 554
13 more
  1. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  2. 2015Unitary Evolution Recurrent Neural Networkscited by 294
  3. 2018Efficient Neural Audio Synthesiscited by 219
  4. 2018On the Practical Computational Power of Finite Precision RNNs for Language Recognitioncited by 117
  5. 2016Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequencescited by 99
  6. 2017Dilated Recurrent Neural Networkscited by 83
  7. 2016Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNscited by 65
  8. 2017Gated Orthogonal Recurrent Units: On Learning to Forgetcited by 31
  9. 2017Recurrent Neural Networks as Weighted Language Recognizerscited by 30
  10. 2024Serial Position Effects of Large Language Modelscited by 30
  11. 2023Primacy Effect of ChatGPTcited by 27
  12. 2023Aspects of human memory and Large Language Modelscited by 12
  13. 2018Learning Long Term Dependencies via Fourier Recurrent Unitscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Fine-grained Analysis of Brain-LLM Alignment through Input Attributioncited by 3
  2. 2025Emergence of Primacy and Recency Effect in Mamba: A Mechanistic Point of Viewcited by 2
  3. 2025Beyond Semantics: How Temporal Biases Shape Retrieval in Transformer and State-Space Modelscited by 0
  4. 2026Extending SSMs with the Exponentially Weighted Signaturecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Memory Determines Learning Direction: A Theory of Gradient-Based Optimization in State Space Models
  2. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing
  3. 2026Sparsity, Superposition, and Forgetting: A Mechanistic Study of Representation Retention in Continual Learning
  4. 2026Temporal Context Reinstatement Drives Episodic-Like Order Memory in Long-Context Language Models
  5. 2024Mimetic Initialization Helps State Space Models Learn to Recall
  6. 2026On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models
  7. 2025Fisher-Guided Selective Forgetting: Mitigating The Primacy Bias in Deep Reinforcement Learning
  8. 2026On the Plasticity Collapse in Continual Machine Unlearning