mlsanityData to 7 Oct 2026

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

arXiv 2402.19427 ↗First posted 29 Feb 2024cs.LG cs.CLcited by 232

Not on the map. Why?

Cited by 43 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recurrent neural networks (RNNs) have fast inference and scale efficiently on long sequences, but they are difficult to train and hard to scale. We propose Hawk, an RNN with gated linear recurrences, and Griffin, a hybrid model that mixes gated linear recurrences with local attention. Hawk exceeds the reported performance of Mamba on downstream tasks, while Griffin matches the performance of Llama-2 despite being trained on over 6 times fewer tokens. We also show that Griffin can extrapolate on sequences significantly longer than those seen during training. Our models match the hardware efficiency of Transformers during training, and during inference they have lower latency and significantly higher throughput. We scale Griffin up to 14B parameters, and explain how to shard our models for efficient distributed training.

Key facts. Cited by 232 papers in the map (data through 7 Oct 2026). Builds on 52 of its 69 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

52 of 69 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2023Mistral 7Bcited by 3,065
38 more
  1. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  4. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  8. 2020GLU Variants Improve Transformercited by 1,724
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  10. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  11. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  12. 2019Root Mean Square Layer Normalizationcited by 1,425
  13. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  14. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  15. 2020On Layer Normalization in the Transformer Architecturecited by 906
  16. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  17. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  18. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  19. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  20. 2016Language Modeling with Gated Convolutional Networkscited by 688
  21. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  22. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  23. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  24. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  25. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  26. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  27. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  28. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  29. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  30. 2022Long Range Language Modeling via Gated State Spacescited by 265
  31. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  32. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  33. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  34. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 126
  35. 2021An Attention Free Transformercited by 111
  36. 2016Quasi-Recurrent Neural Networkscited by 87
  37. 2018NVIDIA Tensor Core Programmability, Performance & Precisioncited by 77
  38. 2024MambaByte: Token-free Selective State Space Modelcited by 58

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 79; 2025: 80; 2026: 73.

232 papers.

Sort:

By area:

  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  2. 2024Titans: Learning to Memorize at Test Timecited by 303
  3. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  4. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  5. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  6. 2024An Empirical Study of Mamba-based Language Modelscited by 177
  7. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  8. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
  9. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  10. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  11. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  12. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
38 more
  1. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 99
  2. 2026Kimi K3: Open Frontier Intelligencecited by 99
  3. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  4. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  5. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
  6. 2024RecurrentGemma: Moving Past Transformers for Efficient Open Language Modelscited by 61
  7. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 60
  8. 2024Longhorn: State Space Models are Amortized Online Learnerscited by 58
  9. 2024Linearizing Large Language Modelscited by 51
  10. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  11. 2024The Expressive Capacity of State Space Models: A Formal Language Perspectivecited by 45
  12. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45
  13. 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 44
  14. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  15. 2024Falcon Mamba: The First Competitive Attention-free 7B Language Modelcited by 43
  16. 2025Learning without training: The implicit dynamics of in-context learningcited by 41
  17. 2024Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRscited by 39
  18. 2024LION: Linear Group RNN for 3D Object Detection in Point Cloudscited by 38
  19. 2025A Systematic Analysis of Hybrid Linear Attentioncited by 38
  20. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 37
  21. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  22. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 36
  23. 2025TAPNext: Tracking Any Point (TAP) as Next Token Predictioncited by 35
  24. 2025How do language models learn facts? Dynamics, curricula and hallucinationscited by 33
  25. 2025Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layerscited by 32
  26. 2024Scalable Event-by-event Processing of Neuromorphic Sensory Signals With Deep State-Space Modelscited by 31
  27. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  28. 2025Hybrid Latent Reasoning via Reinforcement Learningcited by 30
  29. 2024Separations in the Representational Capabilities of Transformers and Recurrent Architecturescited by 29
  30. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 29
  31. 2024Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networkscited by 26
  32. 2024B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memorycited by 26
  33. 2024Recurrent neural networks: vanishing and exploding gradients are not the end of the storycited by 25
  34. 2024LoLCATs: On Low-Rank Linearizing of Large Language Modelscited by 25
  35. 2025The emergence of sparse attention: impact of data distribution and benefits of repetitioncited by 25
  36. 2026Gated DeltaNet-2: Decoupling Erase and Write in Linear Attentioncited by 24
  37. 2024Scalable MatMul-free Language Modelingcited by 21
  38. 2024MetaLA: Unified Optimal Linear Approximation to Softmax Attention Mapcited by 20

50 most cited of 232 All 232 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ParaRNN: Unlocking Parallel Training of Nonlinear RNNs for Large Language Models
  2. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
  3. 2026Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations
  4. 2026M2RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling
  5. 2026Allocating Recurrent Compute in Looped Language Models
  6. 2026AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depth
  7. 2026Closing the Loop: Practical Training Recipes for Looped Language Models
  8. 2024RecurFormer: Not All Transformer Heads Need Self-Attention