mlsanityData to 14 Oct 2026

RWKV-X: A Linear Complexity Hybrid Language Model

arXiv 2504.21463 ↗First posted 30 Apr 2025cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce RWKV-X, a novel hybrid architecture that combines the efficiency of RWKV for short-range modeling with a sparse attention mechanism designed to capture long-range context. Unlike previous hybrid approaches that rely on full attention layers and retain quadratic complexity, RWKV-X achieves linear-time complexity in training and constant-time complexity in inference decoding. We demonstrate that RWKV-X, when continually pretrained on 64K-token sequences, achieves near-perfect accuracy on the 64K passkey retrieval benchmark. It consistently outperforms prior RWKV-7 models on long-context benchmarks, while maintaining strong performance on short-context tasks. These results highlight RWKV-X as a scalable and efficient backbone for general-purpose language modeling, capable of decoding sequences up to 1 million tokens with stable speed and memory usage. To facilitate further research and analysis, we have made the checkpoints and the associated code publicly accessible at: https://github.com/howard-hou/RWKV-X.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 21 of its 23 references that are in the map. First posted on arXiv 30 Apr 2025.

Builds on

21 of 23 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  3. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  6. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  7. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  8. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  9. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  10. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  11. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  12. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 204
9 more
  1. 2025MiniMax-01: Scaling Foundation Models with Lightning Attentioncited by 166
  2. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 159
  3. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  4. 2025RWKV-7 "Goose" with Expressive Dynamic State Evolutioncited by 141
  5. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 124
  6. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 124
  7. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 101
  8. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 75
  9. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 69

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 14
  2. 2025Understanding and Improving Length Generalization in Hierarchical Sparse Attention Modelscited by 6
  3. 2026MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modelingcited by 5
  4. 2026SpikingBrain2.0: Brain-Inspired Foundation Models for Efficient Long-Context and Cross-Platform Inferencecited by 2
  5. 2025Don't Pay Attentioncited by 1
  6. 2026SCRWKV: Ultra-Compact Structure-Calibrated Vision-RWKV for Topological Crack Segmentationcited by 0