Anatomy of Associative Recall in Fixed-State Recurrences: A Matched-State Decomposition, an Interference Wall, and a Curriculum That Breaks It
On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language
Abstract (arXiv)
Fixed-state recurrences--linear attention and state-space models--are reported to lag behind attention on associative recall, but whole-architecture comparisons cannot say which ingredient is responsible. We decompose masked multi-query recall at a fixed state budget along three single-knob axes: a short causal convolution, the transition structure (rank-1 delta rule vs. diagonal), and decay. The convolution dominates (~+0.5 recall in both families under matched training): comparisons that pit convolution-free cells against a convolution-equipped Mamba measure the missing convolution, not the recurrence. The rank-1 transition beats its diagonal ablation by +0.19/+0.32 at 16/32 pairs, but the margin shrinks to +0.03 once both cells carry the convolution, and a state-matched Mamba-2 ties the unarmed rank-1 cell: no class claim survives. Cells that solve 32-pair recall degrade gracefully with load yet fall to chance retrieving 4 pairs from a distractor haystack--flat across lengths and transitions. Interference under sparse supervision, not capacity: a distance curriculum takes the unchanged architecture from 0.021 to 1.000. Training is a lock-in lottery--a seed either locks in or does not--and the curriculum is the lever. Lock-in rises from 1/10 to 7/10 (p=0.02); dense supervision adds nothing; at L=256 a shaped ramp reopens a boundary the uniform curriculum cannot (4/5 vs. 0/9); and at L=512, where the ramp collapses (0/6), gating it on measured accuracy locks in 6/6 (p=0.001). Bidirectional denoiser cells, reading the query before the haystack, show no measurable advantage over causal training (ten seeds), and collision-key retrieval needs two layers. Arming for recall is free on an S_5 state-tracking guardrail--the armed cell is significantly better at every depth (p<=0.0044). These replace "recurrent models are bad at recall" with a measured decomposition and two cheap interventions.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 31 of its 37 references that are in the map. First posted on arXiv 14 Sep 2026.
Builds on
31 of 37 references.
Closest ancestors · 7
Same direction, newest first
- 2025Mechanistic evaluation of Transformers and state space models21 May
- 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues19 Nov
- 2024Just read twice: closing the recall gap for recurrent language models7 Jul
- 2024DeciMamba: Exploring the Length Extrapolation Potential of Mamba20 Jun
- 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
- 2024Simple linear attention language models balance the recall-throughput tradeoff28 Feb
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
Other references · 24
- 2017Attention Is All You Needcited by 59,932
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
- 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
- 2024Simple and Effective Masked Diffusion Language Modelscited by 800
- 2019Designing and Interpreting Probes with Control Taskscited by 608
- 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
- 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
- 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
- 2022Transformers Learn Shortcuts to Automatacited by 260
- 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
- 2022Neural Networks and the Chomsky Hierarchycited by 214
12 more
- 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
- 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 188
- 2024The Illusion of State in State-Space Modelscited by 181
- 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
- 2024The pitfalls of next-token predictioncited by 156
- 2025RWKV-7 "Goose" with Expressive Dynamic State Evolutioncited by 134
- 2025Understanding and Improving Length Generalization in Recurrent Modelscited by 19
- 2025Revisiting associative recall in modern recurrent modelscited by 7
- 2025Quantifying Memory Utilization with Effective State-Sizecited by 5
- 2025DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbonecited by 4
- 2026Echo: KV-Cache-Free Associative Recall with Spectral Koopman Operatorscited by 1
- 2026Triplet-Block Diffusion RWKVcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Test-time regression: a unifying framework for designing sequence models with associative memory
- 2026A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets
- 2026Remembering Distinct Items, Not Tokens: A Learnable Dirichlet-Process Cache Between State-Space Models and Attention
- 2026Memory by Design: Probabilistic Sequence Layers
- 2025LoLA: Low-Rank Linear Attention With Sparse Caching
- 2026RunningTensor: Generalizing Linear Attention to Higher-Order Recurrent States
- 2026SchemaMem: Schema-Indexed Recurrent Memory for Delayed State Retrieval
- 2026MARCH: Scaling Recurrent Memory with Content-Routed State Anchors