# ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language Model Born from Transformer

arXiv: https://arxiv.org/abs/2501.15570 · HTML: https://mlsanity.com/paper/2501.15570/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 13 of its 13 references that are in the map.
- First posted on arXiv 26 Jan 2025.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 13 of 13)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)
- [Parallelizing Linear Transformers with the Delta Rule over Sequence Length](https://mlsanity.com/paper/2406.06484/) (2024)
- [Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing](https://mlsanity.com/paper/2406.08464/) (2024)
- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [The Illusion of State in State-Space Models](https://mlsanity.com/paper/2404.08819/) (2024)
- [Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence](https://mlsanity.com/paper/2404.05892/) (2024)
- [The Mamba in the Llama: Distilling and Accelerating Hybrid Models](https://mlsanity.com/paper/2408.15237/) (2024)
- [Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues](https://mlsanity.com/paper/2411.12537/) (2024)
- [Hymba: A Hybrid-head Architecture for Small Language Models](https://mlsanity.com/paper/2411.13676/) (2024)
- [Does Representation Matter? Exploring Intermediate Layers in Large Language Models](https://mlsanity.com/paper/2412.09563/) (2024)

## Built on by (most cited first, 9 of 9)

- [RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale](https://mlsanity.com/paper/2505.03005/) (2025)
- [Distilling to Hybrid Attention Models via KL-Guided Layer Selection](https://mlsanity.com/paper/2512.20569/) (2025)
- [WuNeng: Hybrid State with Attention](https://mlsanity.com/paper/2504.19191/) (2025)
- [Sliding-window beats linear attention](https://mlsanity.com/paper/2608.28444/) (2026)
- [BlackGoose Rimer: Harnessing RWKV-7 as a Simple yet Superior Replacement for Transformers in Large-Scale Time Series Modeling](https://mlsanity.com/paper/2503.06121/) (2025)
- [What Matters in Linearizing Language Models? A Comparative Study of Architecture, Scale, and Task Adaptation](https://mlsanity.com/paper/2504.14366/) (2025)
- [When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models](https://mlsanity.com/paper/2603.26556/) (2026)
- [RWKVTTS: Yet another TTS based on RWKV-7](https://mlsanity.com/paper/2504.03289/) (2025)
- [State Tuning: State-based Test-Time Scaling on RWKV-7](https://mlsanity.com/paper/2504.05097/) (2025)

## Abstract (arXiv)

As is known, hybrid quadratic and subquadratic attention models in multi-head architectures have surpassed both Transformer and Linear RNN models , with these works primarily focusing on reducing KV complexity and improving efficiency. For further research on expressiveness, we introduce our series of models distilled from Qwen 2.5, based on pure native RWKV-7 attention, which aims to make RNN more expressive and demonstrates state tracking ability beyond transformers. We work with QRWK 32B based on RWKV-6 architecture, another approach that reduces the entire knowledge processing time to just 8 hours using 16 AMD MI300X GPUs while maintaining Qwen 2.5's performance. In fact, the distillation process can utilize any LLM, not just Qwen, and enables knowledge transfer from larger LLMs to smaller ones with more fewer tokens. We will explain the detailed process and share our insights on building more powerful foundation models. Please note that this is an ongoing work that will be updated continuously. The model checkpoints and source code are available at hrefhttps://github.com/yynil/RWKVInsidehttps://github.com/yynil/RWKVInside, hrefhttps://huggingface.co/RWKV-Red-Team/ARWKV-7B-Preview-0.1https://huggingface.co/RWKV-Red-Team/ARWKV-7B-Preview-0.1.