mlsanityData to 7 Oct 2026

Mimetic Initialization Helps State Space Models Learn to Recall

arXiv 2410.11135 ↗First posted 14 Oct 2024cs.LG cs.CLcited by 15

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Recent work has shown that state space models such as Mamba are significantly worse than Transformers on recall-based tasks due to the fact that their state size is constant with respect to their input sequence length. But in practice, state space models have fairly large state sizes, and we conjecture that they should be able to perform much better at these tasks than previously reported. We investigate whether their poor copying and recall performance could be due in part to training difficulties rather than fundamental capacity constraints. Based on observations of their "attention" maps, we propose a structured initialization technique that allows state space layers to more readily mimic attention. Across a variety of architecture settings, our initialization makes it substantially easier for Mamba to learn to copy and do associative recall from scratch.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 19 of its 19 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

19 of 19 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024An Empirical Study of Mamba-based Language Models12 Jun
  2. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  3. 2024The Hidden Attention of Mamba Models3 Mar
  4. 2024Simple linear attention language models balance the recall-throughput tradeoff28 Feb
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  6. 2022Simplified State Space Layers for Sequence Modeling9 Aug
  7. 2022Diagonal State Spaces are as Effective as Structured State Spaces27 Mar
  8. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 11

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  3. 2022In-context Learning and Induction Headscited by 930
  4. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  5. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  6. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  7. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 119
  8. 2024Mechanistic Design and Scaling of Hybrid Architecturescited by 57
  9. 2023Mimetic Initialization of Self-Attention Layerscited by 52
  10. 2023Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priorscited by 33
  11. 2022Understanding the Covariance Structure of Convolutional Filterscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 7.

15 papers.

Sort:

By area:

  1. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
  2. 2025Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layerscited by 32
  3. 2025Multimodal Mamba: Decoder-only Multimodal State Space Model via Quadratic to Linear Distillationcited by 11
  4. 2025Mechanistic evaluation of Transformers and state space modelscited by 8
  5. 2025Revisiting associative recall in modern recurrent modelscited by 7
  6. 2025Fixed-Point RNNs: Interpolating from Diagonal to Densecited by 5
  7. 2025LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Modelscited by 3
  8. 2024Revealing and Mitigating the Local Pattern Shortcuts of Mambacited by 2
  9. 2026Mimetic Initialization of MLPscited by 2
  10. 2026Multi-Mixer Models: Flexible Sequence Modeling with Shared Representationscited by 1
  11. 2026SHiPPO: Recurrent Memory with Transported Polynomial Projectionscited by 1
  12. 2026On the Recall Scaling Laws in Mamba: A Theoretical and Mechanistic Study via Hashingcited by 1
3 more
  1. 2026On the Importance of Gating: Memorization vs. In-Context Learning in State Space Modelscited by 0
  2. 2026Copy the Same, Distill the Difference: Initializing Linear Vision Transformerscited by 0
  3. 2026Adapting Linear-Time Architectures for Tabular In-Context Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding and Enhancing Mamba-Transformer Hybrids for Memory Recall and Language Modeling
  2. 2025Memory Determines Learning Direction: A Theory of Gradient-Based Optimization in State Space Models
  3. 2025Understanding Input Selectivity in Mamba: Impact on Approximation Power, Memorization, and Associative Recall Capacity
  4. 2026Expressivity-Efficiency Tradeoffs for Hybrid Sequence Models
  5. 2025LISM: Long-range Integrative State space Models via Input-Latent State Interactions
  6. 2026Retrieval-Aware Distillation for Transformer-SSM Hybrids
  7. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing
  8. 2025Emergence of Primacy and Recency Effect in Mamba: A Mechanistic Point of View