mlsanityData to 7 Oct 2026

The Evolution of RWKV: Advancements in Efficient Language Modeling

arXiv 2411.02795 ↗First posted 5 Nov 2024cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

This paper reviews the development of the Receptance Weighted Key Value (RWKV) architecture, emphasizing its advancements in efficient language modeling. RWKV combines the training efficiency of Transformers with the inference efficiency of RNNs through a novel linear attention mechanism. We examine its core innovations, adaptations across various domains, and performance advantages over traditional models. The paper also discusses challenges and future directions for RWKV as a versatile architecture in deep learning.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 18 of its 23 references that are in the map. First posted on arXiv 5 Nov 2024.

Builds on

18 of 23 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  7. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  10. 2020Rethinking Attention with Performerscited by 1,584
  11. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  12. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
6 more
  1. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  2. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  3. 2024Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architecturescited by 97
  4. 2024Restore-RWKV: Efficient and Effective Medical Image Restoration with RWKVcited by 36
  5. 2024RWKV-CLIP: A Robust Vision-Language Representation Learnercited by 23
  6. 2024PointRWKV: Efficient RWKV-Like Model for Hierarchical Point Cloud Learningcited by 21

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Linear Attention Modeling for Learned Image Compressioncited by 22