mlsanityData to 14 Oct 2026

RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale

arXiv 2505.03005 ↗First posted 5 May 2025cs.CL cs.AI cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

We present Rapid Attention Distillation to Linear Attention Decoders at Scale (RADLADS), a protocol for rapidly converting softmax attention transformers into linear attention decoder models, along with two new RWKV-variant architectures, and models converted from popular Qwen2.5 open source models in 7B, 32B, and 72B sizes. Our conversion process requires only 350-700M tokens, less than 0.005% of the token count used to train the original teacher models. Converting to our 72B linear attention model costs less than 2,000 USD at today's prices, yet quality at inference remains close to the original transformer. These models achieve state-of-the-art downstream performance across a set of standard benchmarks for linear attention models of their size. We release all our models on HuggingFace under the Apache 2.0 license, with the exception of our 72B models which are also governed by the Qwen License Agreement. Models at https://huggingface.co/collections/recursal/radlads-6818ee69e99e729ba8a87102 Training Code at https://github.com/recursal/RADLADS-paper

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 35 of its 39 references that are in the map. First posted on arXiv 5 May 2025.

Builds on

35 of 39 references.

References · 35

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
23 more
  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  2. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  3. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  4. 2016Sequence-Level Knowledge Distillationcited by 801
  5. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  6. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  7. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  8. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  9. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 204
  10. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 159
  11. 2025RWKV-7 "Goose" with Expressive Dynamic State Evolutioncited by 141
  12. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 122
  13. 2021Finetuning Pretrained Transformers into RNNscited by 96
  14. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 76
  15. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Modelscited by 67
  16. 2024Linearizing Large Language Modelscited by 51
  17. 2025Llamba: Scaling Distilled Recurrent Models for Efficient Language Processingcited by 35
  18. 2025Softpick: No Attention Sink, No Massive Activations with Rectified Softmaxcited by 22
  19. 2024GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compressioncited by 18
  20. 2024DiJiang: Efficient Large Language Models through Compact Kernelizationcited by 16
  21. 2022DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routingcited by 13
  22. 2025ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language Model Born from Transformercited by 9
  23. 2024Cross-Architecture Transfer Learning for Linear-Cost Inference Transformerscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 18.

24 papers.

Sort:

By area:

  1. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 34
  2. 2025Distilling to Hybrid Attention Models via KL-Guided Layer Selectioncited by 20
  3. 2025LoLA: Low-Rank Linear Attention With Sparse Cachingcited by 12
  4. 2025RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decodingcited by 8
  5. 2025Lizard: An Efficient Linearization Framework for Large Language Modelscited by 6
  6. 2026ViT-AdaLA: Adapting Vision Transformers with Linear Attentioncited by 4
  7. 2026Distill-then-Replace: Efficient Task-Specific Hybrid Attention Model Constructioncited by 3
  8. 2026Effective Distillation to Hybrid xLSTM Architecturescited by 3
  9. 2025Apriel-H1: Towards Efficient Enterprise Reasoning Modelscited by 2
  10. 2026Morphing into Hybrid Attention Modelscited by 2
  11. 2026Sliding-window beats linear attentioncited by 2
  12. 2026Residual-Mass Accounting for Partial-KV Decodingcited by 1
12 more
  1. 2026On Subquadratic Architectures: From Applications to Principlescited by 1
  2. 2026Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillationcited by 1
  3. 2026DiD It in 87 Minutes: A Label-Free Softmax-to-Linear Adaptation of Vision Transformers for Object Detectioncited by 1
  4. 2026HEED: Density-Weighted Residual Alignment for Hybrid Vision-Language Model Distillationcited by 0
  5. 2026DASH: Fast Differentiable Architecture Search for Hybrid Attention in Minutes on a Single GPUcited by 0
  6. 2026Attention Amnesia in Hybrid LLMs: When CoT Fine-Tuning Breaks Long-Range Recall, and How to Fix Itcited by 0
  7. 2026TinyCeNN-LM: Quality-Gated Conversion of Pretrained Attention with CeNN-Inspired Cellular-Recurrent Layerscited by 0
  8. 2026On-Policy Attention Linearizationcited by 0
  9. 2026DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Modelscited by 0
  10. 2026Copy the Same, Distill the Difference: Initializing Linear Vision Transformerscited by 0
  11. 2026YANchor-4B: Effective Long-Horizon Reasoning in O(N) Time with O(1) Memorycited by 0
  12. 2026Bridging KV-Cache Quantization and Linear Attention: From Theory to Pretrained Weight Migrationcited by 0