mlsanityData to 7 Oct 2026

Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence

arXiv 2404.05892 ↗First posted 8 Apr 2024cs.CL cs.AIcited by 155

Not on the map. Why?

Cited by 20 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Eagle (RWKV-5) and Finch (RWKV-6), sequence models improving upon the RWKV (RWKV-4) architecture. Our architectural design advancements include multi-headed matrix-valued states and a dynamic recurrence mechanism that improve expressivity while maintaining the inference efficiency characteristics of RNNs. We introduce a new multilingual corpus with 1.12 trillion tokens and a fast tokenizer based on greedy matching for enhanced multilinguality. We trained four Eagle models, ranging from 0.46 to 7.5 billion parameters, and two Finch models with 1.6 and 3.1 billion parameters and find that they achieve competitive performance across a wide variety of benchmarks. We release all our models on HuggingFace under the Apache 2.0 license. Models at: https://huggingface.co/RWKV Training code at: https://github.com/RWKV/RWKV-LM Inference code at: https://github.com/RWKV/ChatRWKV Time-parallel training code at: https://github.com/RWKV/RWKV-infctx-trainer

Key facts. Cited by 155 papers in the map (data through 7 Oct 2026). Builds on 70 of its 97 references that are in the map. First posted on arXiv 8 Apr 2024.

Builds on

70 of 97 references.

References · 70

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2020Longformer: The Long-Document Transformercited by 3,362
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
38 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  2. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  7. 2019Towards VQA Models That Can Readcited by 1,696
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  10. 2020Reformer: The Efficient Transformercited by 1,658
  11. 2020Rethinking Attention with Performerscited by 1,584
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  13. 2018Group Normalizationcited by 1,514
  14. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  15. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  16. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  17. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  18. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  19. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  20. 2020Efficient Transformers: A Surveycited by 784
  21. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  22. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  23. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  24. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  25. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  26. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  27. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  28. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  29. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  30. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  31. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  32. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  33. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  34. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  35. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  36. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  37. 2020Sparse Sinkhorn Attentioncited by 250
  38. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 41; 2025: 73; 2026: 41.

155 papers.

Sort:

By area:

  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  2. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  3. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  4. 2024Titans: Learning to Memorize at Test Timecited by 303
  5. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  6. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  7. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  8. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  9. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  10. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  11. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  12. 2025A Survey on Latent Reasoningcited by 71
38 more
  1. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  2. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
  3. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 60
  4. 2024Longhorn: State Space Models are Amortized Online Learnerscited by 58
  5. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 57
  6. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  7. 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 44
  8. 2024Falcon Mamba: The First Competitive Attention-free 7B Language Modelcited by 43
  9. 2024LinFusion: 1 GPU, 1 Minute, 16K Imagecited by 42
  10. 2025Log-Linear Attentioncited by 39
  11. 2024LION: Linear Group RNN for 3D Object Detection in Point Cloudscited by 38
  12. 2024Restore-RWKV: Efficient and Effective Medical Image Restoration with RWKVcited by 36
  13. 2024A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?cited by 33
  14. 2025Liger: Linearizing Large Language Models to Gated Recurrent Structurescited by 32
  15. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  16. 2025MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connectionscited by 27
  17. 2024Language Models Need Inductive Biases to Count Inductivelycited by 25
  18. 2025RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scalecited by 24
  19. 2025Linear Attention Modeling for Learned Image Compressioncited by 22
  20. 2024PointRWKV: Efficient RWKV-Like Model for Hierarchical Point Cloud Learningcited by 21
  21. 2024Autoregressive Pretraining with Mamba in Visioncited by 21
  22. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 21
  23. 2025Autoregressive Image Generation with Randomized Parallel Decodingcited by 19
  24. 2025Understanding and Improving Length Generalization in Recurrent Modelscited by 19
  25. 2024Mamba or RWKV: Exploring High-Quality and High-Efficiency Segment Anything Modelcited by 18
  26. 2025Lattice: Learning to Efficiently Compress the Memorycited by 18
  27. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothingcited by 17
  28. 2025xLSTM 7B: A Recurrent LLM for Fast and Efficient Inferencecited by 17
  29. 2025Distilling to Hybrid Attention Models via KL-Guided Layer Selectioncited by 17
  30. 2024MobileMamba: Lightweight Multi-Receptive Visual Mamba Networkcited by 16
  31. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  32. 2025NTIRE 2025 Challenge on Image Super-Resolution (x4): Methods and Resultscited by 12
  33. 2026Memory Caching: RNNs with Growing Memorycited by 12
  34. 2024Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulationcited by 11
  35. 2025Multimodal Mamba: Decoder-only Multimodal State Space Model via Quadratic to Linear Distillationcited by 11
  36. 2025Gating is Weighting: Understanding Gated Linear Attention through In-context Learningcited by 11
  37. 2024VisualRWKV: Exploring Recurrent Neural Networks for Visual Language Modelscited by 10
  38. 2025LoLA: Low-Rank Linear Attention With Sparse Cachingcited by 10

50 most cited of 155 All 155 (JSON)