mlsanityData to 7 Oct 2026

RecurrentGemma: Moving Past Transformers for Efficient Open Language Models

arXiv 2404.07839 ↗First posted 11 Apr 2024cs.LG cs.AI cs.CLcited by 61

Not on the map. Why?

Abstract (arXiv)

We introduce RecurrentGemma, a family of open language models which uses Google's novel Griffin architecture. Griffin combines linear recurrences with local attention to achieve excellent performance on language. It has a fixed-sized state, which reduces memory use and enables efficient inference on long sequences. We provide two sizes of models, containing 2B and 9B parameters, and provide pre-trained and instruction tuned variants for both. Our models achieve comparable performance to similarly-sized Gemma baselines despite being trained on fewer tokens.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 5 of its 9 references that are in the map. First posted on arXiv 11 Apr 2024.

Builds on

5 of 9 references.

References · 5

  1. 2020Longformer: The Long-Document Transformercited by 3,362
  2. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  4. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  5. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 25; 2025: 19; 2026: 17.

61 papers.

Sort:

By area:

  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  2. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  3. 2024Titans: Learning to Memorize at Test Timecited by 303
  4. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  5. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  6. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
  7. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
  8. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansioncited by 47
  9. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  10. 2024Falcon Mamba: The First Competitive Attention-free 7B Language Modelcited by 43
  11. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  12. 2025A Systematic Analysis of Hybrid Linear Attentioncited by 38
38 more
  1. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 29
  2. 2025Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptationcited by 25
  3. 2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scalingcited by 21
  4. 2026Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformerscited by 21
  5. 2024Long-Form Speech Generation with Spoken Language Modelscited by 19
  6. 2025xLSTM 7B: A Recurrent LLM for Fast and Efficient Inferencecited by 17
  7. 2024Mamba-PTQ: Outlier Channels in Recurrent Large Language Modelscited by 15
  8. 2025Mamba Modulation: On the Length Generalization of Mambacited by 13
  9. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 13
  10. 2024LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptationcited by 11
  11. 2024The Factuality of Large Language Models in the Legal Domaincited by 10
  12. 2025LoLA: Low-Rank Linear Attention With Sparse Cachingcited by 10
  13. 2024How Well Can a Long Sequence Model Model Long Sequences? Comparing Architechtural Inductive Biases on Long-Context Abilitiescited by 9
  14. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mambacited by 9
  15. 2024Rodimus*: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentionscited by 6
  16. 2024MatMamba: A Matryoshka State Space Modelcited by 6
  17. 2025Understanding and Improving Length Generalization in Hierarchical Sparse Attention Modelscited by 6
  18. 2025Optimizing Large Language Models for ESG Activity Detection in Financial Textscited by 5
  19. 2025Overflow Prevention Enhances Long-Context Recurrent LLMscited by 5
  20. 2026Step-resolved data attribution for looped transformerscited by 5
  21. 2024Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucinationcited by 4
  22. 2026Kwai Summary Attention Technical Reportcited by 4
  23. 2024PARSE-Ego4D: Personal Action Recommendation Suggestions for Egocentric Videoscited by 3
  24. 2025Can Small Language Models Reliably Resist Jailbreak Attacks? A Comprehensive Evaluationcited by 3
  25. 2025The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architecturescited by 3
  26. 2026STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMscited by 3
  27. 2024Pretrained Hybrids with MAD Skillscited by 2
  28. 2025Some Attention is All You Need for Retrievalcited by 2
  29. 2026The Routing and Filtering Structure of Attentioncited by 2
  30. 2024ResumeAtlas: Revisiting Resume Classification with Large-Scale Datasets and Large Language Modelscited by 1
  31. 2024Long-context Protein Language Modeling Using Bidirectional Mamba with Shared Projection Layerscited by 1
  32. 2025DySK-Attn: A Framework for Efficient, Real-Time Knowledge Updating in Large Language Models via Dynamic Sparse Knowledge Attentioncited by 1
  33. 2025LISM: Long-range Integrative State space Models via Input-Latent State Interactionscited by 1
  34. 2025HistoryBankQA: Multilingual Temporal Question Answering on Historical Eventscited by 1
  35. 2026Learning the Signature of Memorization in Autoregressive Language Modelscited by 1
  36. 2026The Mask Is Not the Model: Auditing Prefix Invariance in Attention, State-Space, and Hybrid Sequence Modelscited by 1
  37. 2024Universal In-Context Approximation By Prompting Fully Recurrent Modelscited by 0
  38. 2024SUBLLM: A Novel Efficient Architecture with Token Sequence Subsampling for LLMcited by 0

50 most cited of 61 All 61 (JSON)