mlsanityData to 7 Oct 2026

A Survey of RWKV

arXiv 2412.14847 ↗First posted 19 Dec 2024cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

The Receptance Weighted Key Value (RWKV) model offers a novel alternative to the Transformer architecture, merging the benefits of recurrent and attention-based systems. Unlike conventional Transformers, which depend heavily on self-attention, RWKV adeptly captures long-range dependencies with minimal computational demands. By utilizing a recurrent framework, RWKV addresses some computational inefficiencies found in Transformers, particularly in tasks with long sequences. RWKV has recently drawn considerable attention for its robust performance across multiple domains. Despite its growing popularity, no systematic review of the RWKV model exists. This paper seeks to fill this gap as the first comprehensive review of the RWKV architecture, its core principles, and its varied applications, such as natural language generation, natural language understanding, and computer vision. We assess how RWKV compares to traditional Transformer models, highlighting its capability to manage long sequences efficiently and lower computational costs. Furthermore, we explore the challenges RWKV encounters and propose potential directions for future research and advancement. We consistently maintain the related open-source materials at: https://github.com/MLGroupJLU/RWKV-Survey.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 87 of its 198 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

87 of 198 references.

References · 87

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  11. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
38 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  2. 2020Rethinking Attention with Performerscited by 1,584
  3. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  5. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  6. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  7. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  8. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  9. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  10. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  11. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 373
  12. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  13. 2024xLSTM: Extended Long Short-Term Memorycited by 320
  14. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  15. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  16. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  17. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  18. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  19. 2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 167
  20. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  21. 2019Self-Driving Cars: A Surveycited by 147
  22. 2022A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Modelscited by 128
  23. 2021Deep Subdomain Adaptation Network for Image Classificationcited by 119
  24. 2021An Attention Free Transformercited by 111
  25. 2020Dual Mixup Regularized Learning for Adversarial Domain Adaptationcited by 99
  26. 2024Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architecturescited by 97
  27. 2023CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?cited by 89
  28. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  29. 2024In-Context Language Learning: Architectures and Algorithmscited by 86
  30. 2024From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examplescited by 55
  31. 2023SuperCLUE: A Comprehensive Chinese Large Language Model Benchmarkcited by 50
  32. 2024A Survey on Visual Mambacited by 50
  33. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  34. 2024Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Modelscited by 41
  35. 2024LION: Linear Group RNN for 3D Object Detection in Point Cloudscited by 38
  36. 2024Restore-RWKV: Efficient and Effective Medical Image Restoration with RWKVcited by 36
  37. 2024Were RNNs All We Needed?cited by 36
  38. 2024PRE: A Peer Review Based Large Language Model Evaluatorcited by 28

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mambacited by 9
  2. 2025RWKVQuant: Quantizing the RWKV Family with Proxy Guided Hybrid of Scalar and Vector Quantizationcited by 4
  3. 2025EvRWKV: A Continuous Interactive RWKV Framework for Effective Event-Guided Low-Light Image Enhancementcited by 2
  4. 2025Don't Pay Attentioncited by 1
  5. 2026SCRWKV: Ultra-Compact Structure-Calibrated Vision-RWKV for Topological Crack Segmentationcited by 0