mlsanityData to 14 Oct 2026

RWKV: Reinventing RNNs for the Transformer Era

arXiv 2305.13048 ↗First posted 22 May 2023cs.CL cs.AIcited by 827

Not on the map. Why?

Cited by 101 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformers have revolutionized almost all natural language processing (NLP) tasks but suffer from memory and computational complexity that scales quadratically with sequence length. In contrast, recurrent neural networks (RNNs) exhibit linear scaling in memory and computational requirements but struggle to match the same performance as Transformers due to limitations in parallelization and scalability. We propose a novel model architecture, Receptance Weighted Key Value (RWKV), that combines the efficient parallelizable training of transformers with the efficient inference of RNNs. Our approach leverages a linear attention mechanism and allows us to formulate the model as either a Transformer or an RNN, thus parallelizing computations during training and maintains constant computational and memory complexity during inference. We scale our models as large as 14 billion parameters, by far the largest dense RNN ever trained, and find RWKV performs on par with similarly sized Transformers, suggesting future work can leverage this architecture to create more efficient models. This work presents a significant step towards reconciling trade-offs between computational efficiency and model performance in sequence processing tasks.

Key facts. Cited by 827 papers in the map (data through 14 Oct 2026). Builds on 77 of its 108 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

77 of 108 references.

References · 77

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2016Layer Normalizationcited by 5,669
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  12. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
38 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2020Longformer: The Long-Document Transformercited by 3,413
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  7. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,919
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  10. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  12. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  13. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  14. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  15. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  16. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  17. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  18. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  19. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  20. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  21. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  22. 2020Reformer: The Efficient Transformercited by 1,668
  23. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,476
  24. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  25. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  26. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  27. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  28. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  29. 2020Efficient Transformers: A Surveycited by 790
  30. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  31. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  32. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  33. 2018Deep learning for time series classification: a reviewcited by 576
  34. 2021MetaFormer Is Actually What You Need for Visioncited by 524
  35. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  36. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  37. 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 486
  38. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 417

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 59; 2024: 302; 2025: 238; 2026: 228.

827 papers.

Sort:

By area:

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  2. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  3. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  4. 2024VMamba: Visual State Space Modelcited by 969
  5. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 681
  6. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  7. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  8. 2024Large Language Models: A Surveycited by 525
  9. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  10. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  11. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  12. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
38 more
  1. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  2. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  3. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 355
  4. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 328
  5. 2024Titans: Learning to Memorize at Test Timecited by 328
  6. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  7. 2025Memory in the Age of AI Agentscited by 303
  8. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  9. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  10. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  11. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  12. 2023Challenges and Applications of Large Language Modelscited by 254
  13. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 246
  14. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
  15. 2024Foundation Models for Time Series Analysis: A Tutorial and Surveycited by 239
  16. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 235
  17. 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 225
  18. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  19. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 216
  20. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  21. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  22. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  23. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  24. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  25. 2023Efficient Large Language Models: A Surveycited by 186
  26. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  27. 2024U-KAN Makes Strong Backbone for Medical Image Segmentation and Generationcited by 179
  28. 2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 168
  29. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 159
  30. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  31. 2024MambaOut: Do We Really Need Mamba for Vision?cited by 143
  32. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  33. 2024Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spacescited by 134
  34. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  35. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 127
  36. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 124
  37. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 124
  38. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 123

50 most cited of 827 All 827 (JSON)