mlsanityData to 7 Oct 2026

Simple linear attention language models balance the recall-throughput tradeoff

arXiv 2402.18668 ↗First posted 28 Feb 2024cs.CL cs.LGcited by 197

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Recent work has shown that attention-based language models excel at recall, the ability to ground generations in tokens previously seen in context. However, the efficiency of attention-based models is bottle-necked during inference by the KV-cache's aggressive memory consumption. In this work, we explore whether we can improve language model efficiency (e.g. by reducing memory consumption) without compromising on recall. By applying experiments and theory to a broad set of architectures, we identify a key tradeoff between a model's state size and recall ability. We show that efficient alternatives to attention (e.g. H3, Mamba, RWKV) maintain a fixed-size recurrent state, but struggle at recall. We propose BASED a simple architecture combining linear and sliding window attention. By varying BASED window size and linear attention feature dimension, we can dial the state size and traverse the pareto frontier of the recall-memory tradeoff curve, recovering the full quality of attention on one end and the small state size of attention-alternatives on the other. We train language models up to 1.3b parameters and show that BASED matches the strongest sub-quadratic models (e.g. Mamba) in perplexity and outperforms them on real-world recall-intensive tasks by 6.22 accuracy points. Implementations of linear attention are often less efficient than optimized standard attention implementations. To make BASED competitive, we develop IO-aware algorithms that enable 24x higher throughput on language generation than FlashAttention-2, when generating 1024 tokens using 1.3b parameter models. Code for this work is provided at: https://github.com/HazyResearch/based.

Key facts. Cited by 197 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). First posted on arXiv 28 Feb 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 41; 2025: 67; 2026: 89.

197 papers.

Sort:

By area:

  1. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  2. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  3. 2024Titans: Learning to Memorize at Test Timecited by 303
  4. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  5. 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 206
  6. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  7. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  8. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  9. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  10. 2025Test-Time Training Done Rightcited by 121
  11. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  12. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
38 more
  1. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 76
  2. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  3. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  4. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
  5. 2024Longhorn: State Space Models are Amortized Online Learnerscited by 58
  6. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 57
  7. 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 44
  8. 2024Hydra: Bidirectional State Space Models Through Generalized Matrix Mixerscited by 43
  9. 2024Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRscited by 39
  10. 2025Log-Linear Attentioncited by 39
  11. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 37
  12. 2024ThunderKittens: Simple, Fast, and Adorable AI Kernelscited by 34
  13. 2025Liger: Linearizing Large Language Models to Gated Recurrent Structurescited by 32
  14. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 32
  15. 2024A Survey of Small Language Modelscited by 31
  16. 2024Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networkscited by 26
  17. 2024B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memorycited by 26
  18. 2024LoLCATs: On Low-Rank Linearizing of Large Language Modelscited by 25
  19. 2026Gated DeltaNet-2: Decoupling Erase and Write in Linear Attentioncited by 24
  20. 2025Adaptive LLM Routing under Budget Constraintscited by 21
  21. 2024MetaLA: Unified Optimal Linear Approximation to Softmax Attention Mapcited by 20
  22. 2025Understanding and Improving Length Generalization in Recurrent Modelscited by 19
  23. 2025Lattice: Learning to Efficiently Compress the Memorycited by 18
  24. 2025Distilling to Hybrid Attention Models via KL-Guided Layer Selectioncited by 17
  25. 2024Selective Attention: Enhancing Transformer through Principled Context Controlcited by 16
  26. 2026PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformerscited by 16
  27. 2024Mimetic Initialization Helps State Space Models Learn to Recallcited by 15
  28. 2025Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanismcited by 15
  29. 2024ViG: Linear-complexity Visual Sequence Learning with Gated Linear Attentioncited by 13
  30. 2024Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusterscited by 13
  31. 2025Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformerscited by 12
  32. 2025Scaling Linear Attention with Sparse State Expansioncited by 12
  33. 2026Memory Caching: RNNs with Growing Memorycited by 12
  34. 2025Gating is Weighting: Understanding Gated Linear Attention through In-context Learningcited by 11
  35. 2025Short window attention enables long-term memorizationcited by 11
  36. 2025LoLA: Low-Rank Linear Attention With Sparse Cachingcited by 10
  37. 2025RATTENTION: Towards the Minimal Sliding Window Size in Local-Global Attention Modelscited by 10
  38. 2025Native Hybrid Attention for Efficient Sequence Modelingcited by 10

50 most cited of 197 All 197 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Sliding-window beats linear attention
  2. 2026Balancing Memory Pathways: Analyzing and Improving Memory Utilization in Hybrid LMs
  3. 2024Beyond KV Caching: Shared Attention for Efficient LLMs
  4. 2026On-Demand Attention: Language Models Know When to Recall
  5. 2026Rethinking the Role of Efficient Attention in Hybrid Architectures
  6. 2026What Attention Recalls and Recurrence Controls in Hybrid Language Models
  7. 2026RunningTensor: Generalizing Linear Attention to Higher-Order Recurrent States
  8. 2024Just read twice: closing the recall gap for recurrent language models