mlsanityData to 7 Oct 2026

RWKV-7 "Goose" with Expressive Dynamic State Evolution

arXiv 2503.14456 ↗First posted 18 Mar 2025cs.CL cs.AI cs.LGcited by 134

Not on the map. Why?

Abstract (arXiv)

We present RWKV-7 "Goose", a new sequence modeling architecture with constant memory usage and constant inference time per token. Despite being trained on dramatically fewer tokens than other top models, our 2.9 billion parameter language model achieves a new 3B SoTA on multilingual tasks and matches the current 3B SoTA on English language downstream performance. RWKV-7 introduces a newly generalized formulation of the delta rule with vector-valued gating and in-context learning rates, as well as a relaxed value replacement rule. We show that RWKV-7 can perform state tracking and recognize all regular languages, while retaining parallelizability of training. This exceeds the capabilities of Transformers under standard complexity conjectures, which are limited to mathsfTC0. To demonstrate RWKV-7's language modeling capability, we also present an extended open source 3.1 trillion token multilingual corpus, and train four RWKV-7 models ranging from 0.19 billion to 2.9 billion parameters on this dataset. To foster openness, reproduction, and adoption, we release our models and dataset component listing at https://huggingface.co/RWKV, and our training and inference code at https://github.com/RWKV/RWKV-LM all under the Apache 2.0 License.

Key facts. Cited by 134 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Mar 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 57; 2026: 77.

134 papers.

Sort:

By area:

  1. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  2. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  3. 2026Kimi K3: Open Frontier Intelligencecited by 99
  4. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  5. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 60
  6. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 57
  7. 2025Glyph: Scaling Context Windows via Visual-Text Compressioncited by 49
  8. 2025Log-Linear Attentioncited by 39
  9. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  10. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 32
  11. 2025Understanding Transformer from the Perspective of Associative Memorycited by 31
  12. 2025RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scalecited by 24
38 more
  1. 2025Evolution Strategies at the Hyperscalecited by 22
  2. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 21
  3. 2025Lattice: Learning to Efficiently Compress the Memorycited by 18
  4. 2025Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Modelscited by 16
  5. 2025TempoPFN: Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecastingcited by 16
  6. 2026Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contextscited by 16
  7. 2025Structured Sparse Transition Matrices to Enable State Tracking in State-Space Modelscited by 15
  8. 2025Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Modelscited by 15
  9. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 13
  10. 2025Comba: Improving Bilinear RNNs with Closed-loop Controlcited by 12
  11. 2025Scaling Linear Attention with Sparse State Expansioncited by 12
  12. 2026Fast-weight Product Key Memorycited by 11
  13. 2026Test-Time Training with KV Binding Is Secretly Linear Attentioncited by 10
  14. 2025AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understandingcited by 9
  15. 2025Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regressioncited by 9
  16. 2025InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Modelscited by 9
  17. 2025RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decodingcited by 8
  18. 2025Sequential-Parallel Duality in Prefix Scannable Modelscited by 8
  19. 2025Selective Rotary Position Embeddingcited by 8
  20. 2025TNT: Improving Chunkwise Training for Test-Time Memorizationcited by 7
  21. 2026Why Are Linear RNNs More Parallelizable?cited by 7
  22. 2025RWKV-X: A Linear Complexity Hybrid Language Modelcited by 6
  23. 2025On the Expressiveness of Softmax Attention: A Recurrent Neural Network Perspectivecited by 6
  24. 2025Fast weight programming and linear transformers: from machine learning to neurobiologycited by 6
  25. 2026Learning State-Tracking from Code Using Linear RNNscited by 5
  26. 2026Rethinking the Role of Efficient Attention in Hybrid Architecturescited by 5
  27. 2025Beyond Memorization: Extending Reasoning Depth with Recurrence, Memory and Test-Time Compute Scalingcited by 4
  28. 2025Local Linear Attention: An Optimal Interpolation of Linear and Softmax Attention For Test-Time Regressioncited by 4
  29. 2025Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridgecited by 4
  30. 2025The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architecturescited by 3
  31. 2025Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Evictioncited by 3
  32. 2026Effective Distillation to Hybrid xLSTM Architecturescited by 3
  33. 2026Beyond Quadratic: Linear-Time Change Detection with RWKVcited by 3
  34. 2026The Topological Trouble With Transformerscited by 3
  35. 2026MDN: Parallelizing Stepwise Momentum for Delta Linear Attentioncited by 3
  36. 2026Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attentioncited by 3
  37. 2025WuNeng: Hybrid State with Attentioncited by 2
  38. 2025DualComp: End-to-End Learning of a Unified Dual-Modality Lossless Compressorcited by 2

50 most cited of 134 All 134 (JSON)