mlsanityData to 7 Oct 2026

Repeat After Me: Transformers are Better than State Space Models at Copying

arXiv 2402.01032 ↗First posted 1 Feb 2024cs.LG cs.AI cs.CLcited by 208

Not on the map. Why?

Cited by 35 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformers are the dominant architecture for sequence modeling, but there is growing interest in models that use a fixed-size latent state that does not depend on the sequence length, which we refer to as "generalized state space models" (GSSMs). In this paper we show that while GSSMs are promising in terms of inference-time efficiency, they are limited compared to transformer models on tasks that require copying from the input context. We start with a theoretical analysis of the simple task of string copying and prove that a two layer transformer can copy strings of exponential length while GSSMs are fundamentally limited by their fixed-size latent state. Empirically, we find that transformers outperform GSSMs in terms of efficiency and generalization on synthetic tasks that require copying the context. Finally, we evaluate pretrained large language models and find that transformer models dramatically outperform state space models at copying and retrieving information from context. Taken together, these results suggest a fundamental gap between transformers and GSSMs on tasks of practical interest.

Key facts. Cited by 208 papers in the map (data through 7 Oct 2026). Builds on 38 of its 51 references that are in the map. First posted on arXiv 1 Feb 2024.

Builds on

38 of 51 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  9. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
26 more
  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  2. 2020Rethinking Attention with Performerscited by 1,584
  3. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  4. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  5. 2022In-context Learning and Induction Headscited by 930
  6. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  7. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  8. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  9. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  10. 2022Neural Networks and the Chomsky Hierarchycited by 214
  11. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  12. 2023HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolutioncited by 187
  13. 2020How Context Affects Language Models' Factual Predictionscited by 181
  14. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  15. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 137
  16. 2023Representational Strengths and Limitations of Transformerscited by 132
  17. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  18. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  19. 2016Quasi-Recurrent Neural Networkscited by 87
  20. 2024In-Context Language Learning: Architectures and Algorithmscited by 86
  21. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74
  22. 2023Tighter Bounds on the Expressivity of Transformer Encoderscited by 65
  23. 2023Length Generalization in Arithmetic Transformerscited by 64
  24. 2020A Formal Hierarchy of RNN Architecturescited by 61
  25. 2023Positional Description Matters for Transformers Arithmeticcited by 56
  26. 2019Sequential Neural Networks as Automatacited by 54

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 56; 2025: 79; 2026: 73.

208 papers.

Sort:

By area:

  1. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  2. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  3. 2024An Empirical Study of Mamba-based Language Modelscited by 177
  4. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  5. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  6. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  7. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 138
  8. 2025TTT3R: 3D Reconstruction as Test-Time Trainingcited by 122
  9. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  10. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  11. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  12. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
38 more
  1. 2024Theoretical Foundations of Deep Selective State-Space Modelscited by 73
  2. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
  3. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
  4. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 68
  5. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  6. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  7. 2024RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrievalcited by 62
  8. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  9. 2024Larimar: Large Language Models with Episodic Memory Controlcited by 53
  10. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  11. 2024The Expressive Capacity of State Space Models: A Formal Language Perspectivecited by 45
  12. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  13. 2024Hydra: Bidirectional State Space Models Through Generalized Matrix Mixerscited by 43
  14. 2024Theoretical limitations of multi-layer Transformercited by 43
  15. 2024A Survey of Mambacited by 39
  16. 2025A Systematic Analysis of Hybrid Linear Attentioncited by 38
  17. 2024MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memorycited by 35
  18. 2025Llamba: Scaling Distilled Recurrent Models for Efficient Language Processingcited by 35
  19. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 32
  20. 2025Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layerscited by 32
  21. 2024Separations in the Representational Capabilities of Transformers and Recurrent Architecturescited by 29
  22. 2024Universal Length Generalization with Turing Programscited by 25
  23. 2026Gated DeltaNet-2: Decoupling Erase and Write in Linear Attentioncited by 24
  24. 2024A Hitchhiker's Guide to Scaling Law Estimationcited by 23
  25. 2025Understanding and Improving Length Generalization in Recurrent Modelscited by 19
  26. 2025Long-Context Generalization with Sparse Attentioncited by 18
  27. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothingcited by 17
  28. 2026Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contextscited by 16
  29. 2024Associative Recurrent Memory Transformercited by 15
  30. 2024Mimetic Initialization Helps State Space Models Learn to Recallcited by 15
  31. 2025Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanismcited by 15
  32. 2025Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routingcited by 13
  33. 2025Mamba Modulation: On the Length Generalization of Mambacited by 13
  34. 2025The Bayesian Geometry of Transformer Attentioncited by 13
  35. 2025Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scalecited by 12
  36. 2025Scaling Linear Attention with Sparse State Expansioncited by 12
  37. 2024Expansion Span: Combining Fading Memory and Retrieval in Hybrid State Space Modelscited by 11
  38. 2025Gating is Weighting: Understanding Gated Linear Attention through In-context Learningcited by 11

50 most cited of 208 All 208 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D
  2. 2022Efficient Long Sequence Modeling via State Space Augmented Transformer
  3. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Models
  4. 2024On the Expressiveness and Length Generalization of Selective State-Space Models on Regular Languages
  5. 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving
  6. 2024CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling
  7. 2026Mamba-3: Improved Sequence Modeling using State Space Principles
  8. 2026The Surprising Effectiveness of Shared Memory in Looped Transformers