mlsanityData to 14 Oct 2026

Zoology: Measuring and Improving Recall in Efficient Language Models

arXiv 2312.04927 ↗First posted 8 Dec 2023cs.CL cs.LGcited by 204

Not on the map. Why?

Cited by 30 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Attention-free language models that combine gating and convolutions are growing in popularity due to their efficiency and increasingly competitive performance. To better understand these architectures, we pretrain a suite of 17 attention and "gated-convolution" language models, finding that SoTA gated-convolution architectures still underperform attention by up to 2.1 perplexity points on the Pile. In fine-grained analysis, we find 82% of the gap is explained by each model's ability to recall information that is previously mentioned in-context, e.g. "Hakuna Matata means no worries Hakuna Matata it means no" rightarrow "??". On this task, termed "associative recall", we find that attention outperforms gated-convolutions by a large margin: a 70M parameter attention model outperforms a 1.4 billion parameter gated-convolution model on associative recall. This is surprising because prior work shows gated convolutions can perfectly solve synthetic tests for AR capability. To close the gap between synthetics and real language, we develop a new formalization of the task called multi-query associative recall (MQAR) that better reflects actual language. We perform an empirical and theoretical study of MQAR that elucidates differences in the parameter-efficiency of attention and gated-convolution recall. Informed by our analysis, we evaluate simple convolution-attention hybrids and show that hybrids with input-dependent sparse attention patterns can close 97.4% of the gap to attention, while maintaining sub-quadratic scaling. Our code is accessible at: https://github.com/HazyResearch/zoology.

Key facts. Cited by 204 papers in the map (data through 14 Oct 2026). First posted on arXiv 8 Dec 2023.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 51; 2025: 55; 2026: 97.

204 papers.

Sort:

By area:

  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  2. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  3. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  4. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  5. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 235
  6. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  7. 2024An Empirical Study of Mamba-based Language Modelscited by 186
  8. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 174
  9. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  10. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 127
  11. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 126
  12. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 122
38 more
  1. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 105
  2. 2024In-Context Language Learning: Architectures and Algorithmscited by 89
  3. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 86
  4. 2024Theoretical Foundations of Deep Selective State-Space Modelscited by 74
  5. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 74
  6. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 69
  7. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 66
  8. 2024RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrievalcited by 64
  9. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMscited by 64
  10. 2024MambaByte: Token-free Selective State Space Modelcited by 59
  11. 2024Mechanistic Design and Scaling of Hybrid Architecturescited by 59
  12. 2024Longhorn: State Space Models are Amortized Online Learnerscited by 59
  13. 2024Jamba-1.5: Hybrid Transformer-Mamba Models at Scalecited by 50
  14. 2024Hydra: Bidirectional State Space Models Through Generalized Matrix Mixerscited by 44
  15. 2025Log-Linear Attentioncited by 40
  16. 2024BlackMamba: Mixture of Experts for State-Space Modelscited by 37
  17. 2025How do language models learn facts? Dynamics, curricula and hallucinationscited by 37
  18. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 34
  19. 2026Gated DeltaNet-2: Decoupling Erase and Write in Linear Attentioncited by 33
  20. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 31
  21. 2024Separations in the Representational Capabilities of Transformers and Recurrent Architecturescited by 29
  22. 2024Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networkscited by 26
  23. 2024Understanding Transformers via N-gram Statisticscited by 26
  24. 2025The emergence of sparse attention: impact of data distribution and benefits of repetitioncited by 25
  25. 2025RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scalecited by 24
  26. 2024MetaLA: Unified Optimal Linear Approximation to Softmax Attention Mapcited by 21
  27. 2025Long-Context Generalization with Sparse Attentioncited by 21
  28. 2025Distilling to Hybrid Attention Models via KL-Guided Layer Selectioncited by 20
  29. 2024LoCoCo: Dropping In Convolutions for Long Context Compressioncited by 19
  30. 2025Lattice: Learning to Efficiently Compress the Memorycited by 19
  31. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothingcited by 18
  32. 2024An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encodingcited by 16
  33. 2025Mamba Modulation: On the Length Generalization of Mambacited by 16
  34. 2025Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanismcited by 15
  35. 2025Comba: Improving Bilinear RNNs with Closed-loop Controlcited by 15
  36. 2025ParaRNN: Unlocking Parallel Training of Nonlinear RNNs for Large Language Modelscited by 15
  37. 2026Memory Caching: RNNs with Growing Memorycited by 15
  38. 2025Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routingcited by 14

50 most cited of 204 All 204 (JSON)