mlsanityData to 7 Oct 2026

Hungry Hungry Hippos: Towards Language Modeling with State Space Models

arXiv 2212.14052 ↗First posted 28 Dec 2022cs.LG cs.CLcited by 495

Not on the map. Why?

Cited by 194 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

State space models (SSMs) have demonstrated state-of-the-art sequence modeling performance in some modalities, but underperform attention in language modeling. Moreover, despite scaling nearly linearly in sequence length instead of quadratically, SSMs are still slower than Transformers due to poor hardware utilization. In this paper, we make progress on understanding the expressivity gap between SSMs and attention in language modeling, and on reducing the hardware barrier between SSMs and attention. First, we use synthetic language modeling tasks to understand the gap between SSMs and attention. We find that existing SSMs struggle with two capabilities: recalling earlier tokens in the sequence and comparing tokens across the sequence. To understand the impact on language modeling, we propose a new SSM layer, H3, that is explicitly designed for these abilities. H3 matches attention on the synthetic languages and comes within 0.4 PPL of Transformers on OpenWebText. Furthermore, a hybrid 125M-parameter H3-attention model that retains two attention layers surprisingly outperforms Transformers on OpenWebText by 1.0 PPL. Next, to improve the efficiency of training SSMs on modern hardware, we propose FlashConv. FlashConv uses a fused block FFT algorithm to improve efficiency on sequences up to 8K, and introduces a novel state passing algorithm that exploits the recurrent properties of SSMs to scale to longer sequences. FlashConv yields 2× speedup on the long-range arena benchmark and allows hybrid language models to generate text 2.4× faster than Transformers. Using FlashConv, we scale hybrid H3-attention language models up to 2.7B parameters on the Pile and find promising initial results, achieving lower perplexity than Transformers and outperforming Transformers in zero- and few-shot learning on a majority of tasks in the SuperGLUE benchmark.

Key facts. Cited by 495 papers in the map (data through 7 Oct 2026). Builds on 41 of its 65 references that are in the map. First posted on arXiv 28 Dec 2022.

Builds on

41 of 65 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
29 more
  1. 2020Reformer: The Efficient Transformercited by 1,658
  2. 2020Rethinking Attention with Performerscited by 1,584
  3. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  4. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  5. 2022In-context Learning and Induction Headscited by 930
  6. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  7. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  8. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  9. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  10. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  11. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  12. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  13. 2022Long Range Language Modeling via Gated State Spacescited by 265
  14. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  15. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  16. 2016Using Fast Weights to Attend to the Recent Pastcited by 161
  17. 2020The Hardware Lotterycited by 160
  18. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  19. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  20. 2022On The Computational Complexity of Self-Attentioncited by 137
  21. 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projectionscited by 121
  22. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 100
  23. 2018The Temple University Hospital Seizure Detection Corpuscited by 80
  24. 2021Self-Supervised Graph Neural Networks for Improved Electroencephalographic Seizure Analysiscited by 75
  25. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  26. 2021Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encodingcited by 50
  27. 2020SMYRF: Efficient Attention using Asymmetric Clusteringcited by 43
  28. 2022Self-Supervised Learning of Brain Dynamics from Broad Neuroimaging Datacited by 39
  29. 2020Fine-grain atlases of functional modes for fMRI analysiscited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 57; 2024: 225; 2025: 125; 2026: 88.

495 papers.

Sort:

By area:

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  2. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  3. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  4. 2024VMamba: Visual State Space Modelcited by 955
  5. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  6. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  7. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  8. 2023The Falcon Series of Open Language Modelscited by 526
  9. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  10. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  11. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  12. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
38 more
  1. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  2. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  3. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  4. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  5. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 325
  6. 2024Titans: Learning to Memorize at Test Timecited by 303
  7. 2023Challenges and Applications of Large Language Modelscited by 251
  8. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  9. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  10. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  11. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  12. 2023HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolutioncited by 187
  13. 2023Efficient Large Language Models: A Surveycited by 183
  14. 2024The Illusion of State in State-Space Modelscited by 181
  15. 2024U-KAN Makes Strong Backbone for Medical Image Segmentation and Generationcited by 176
  16. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  17. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  18. 2024PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognitioncited by 141
  19. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  20. 2024Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spacescited by 132
  21. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  22. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  23. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 119
  24. 2024ChangeMamba: Remote Sensing Change Detection With Spatiotemporal State Space Modelcited by 118
  25. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  26. 2024MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detectioncited by 113
  27. 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 108
  28. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 104
  29. 2024RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulationcited by 102
  30. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  31. 2024Demystify Mamba in Vision: A Linear Attention Perspectivecited by 101
  32. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  33. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inferencecited by 97
  34. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 97
  35. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  36. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  37. 2024In-Context Language Learning: Architectures and Algorithmscited by 86
  38. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84

50 most cited of 495 All 495 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024An Empirical Study of Mamba-based Language Models
  2. 2022Efficient Long Sequence Modeling via State Space Augmented Transformer
  3. 2026Interpreting and Steering State-Space Models via Activation Subspace Bottlenecks
  4. 2023A Quantitative Review on Language Model Efficiency Research
  5. 2024LongSSM: On the Length Extension of State-space Models in Language Modelling
  6. 2024How Effective are State Space Models for Machine Translation?
  7. 2025Implicit Language Models are RNNs: Balancing Parallelization and Expressivity
  8. 2025Characterizing Mamba's Selective Memory using Auto-Encoders