# Emergence of Primacy and Recency Effect in Mamba: A Mechanistic Point of View

arXiv: https://arxiv.org/abs/2506.15156 · HTML: https://mlsanity.com/paper/2506.15156/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 8 of its 15 references that are in the map.
- First posted on arXiv 18 Jun 2025.

## Placed in

- State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/)

## Builds on (most cited first, 8 of 8)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Repeat After Me: Transformers are Better than State Space Models at Copying](https://mlsanity.com/paper/2402.01032/) (2024)
- [On the Emergence of Position Bias in Transformers](https://mlsanity.com/paper/2502.01951/) (2025)
- [Falcon Mamba: The First Competitive Attention-free 7B Language Model](https://mlsanity.com/paper/2410.05355/) (2024)
- [Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing](https://mlsanity.com/paper/2501.00658/) (2024)
- [Emergence of the Primacy Effect in Structured State-Space Models](https://mlsanity.com/paper/2502.13729/) (2025)

## Built on by (most cited first, 2 of 2)

- [WriteSAE: Sparse Autoencoders for Recurrent State](https://mlsanity.com/paper/2605.12770/) (2026)
- [Beyond Semantics: How Temporal Biases Shape Retrieval in Transformer and State-Space Models](https://mlsanity.com/paper/2510.22752/) (2025)

## Abstract (arXiv)

We study memory in state-space language models using primacy and recency effects as behavioral tools to uncover how information is retained and forgotten over time. Applying structured recall tasks to the Mamba architecture, we observe a consistent U-shaped accuracy profile, indicating strong performance at the beginning and end of input sequences. We identify three mechanisms that give rise to this pattern. First, long-term memory is supported by a sparse subset of channels within the model's selective state space block, which persistently encode early input tokens and are causally linked to primacy effects. Second, short-term memory is governed by delta-modulated recurrence: recent inputs receive more weight due to exponential decay, but this recency advantage collapses when distractor items are introduced, revealing a clear limit to memory depth. Third, we find that memory allocation is dynamically modulated by semantic regularity: repeated relations in the input sequence shift the delta gating behavior, increasing the tendency to forget intermediate items. We validate these findings via targeted ablations and input perturbations on two large-scale Mamba-based language models: one with 1.4B and another with 7B parameters.