# Distilling to Hybrid Attention Models via KL-Guided Layer Selection

arXiv: https://arxiv.org/abs/2512.20569 · HTML: https://mlsanity.com/paper/2512.20569/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 46 references that are in the map.
- First posted on arXiv 23 Dec 2025.

## Builds on (most cited first, 30 of 39)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://mlsanity.com/paper/2412.06464/) (2024)
- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [Parallelizing Linear Transformers with the Delta Rule over Sequence Length](https://mlsanity.com/paper/2406.06484/) (2024)
- [Simple linear attention language models balance the recall-throughput tradeoff](https://mlsanity.com/paper/2402.18668/) (2024)
- [Zoology: Measuring and Improving Recall in Efficient Language Models](https://mlsanity.com/paper/2312.04927/) (2023)
- [Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling](https://mlsanity.com/paper/2406.07522/) (2024)
- [Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence](https://mlsanity.com/paper/2404.05892/) (2024)
- [HGRN2: Gated Linear RNNs with State Expansion](https://mlsanity.com/paper/2404.07904/) (2024)
- [The Mamba in the Llama: Distilling and Accelerating Hybrid Models](https://mlsanity.com/paper/2408.15237/) (2024)
- [Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues](https://mlsanity.com/paper/2411.12537/) (2024)
- [Finetuning Pretrained Transformers into RNNs](https://mlsanity.com/paper/2103.13076/) (2021)
- [Gated Slot Attention for Efficient Linear-Time Sequence Modeling](https://mlsanity.com/paper/2409.07146/) (2024)
- [DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products](https://mlsanity.com/paper/2502.10297/) (2025)
- [RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval](https://mlsanity.com/paper/2402.18510/) (2024)
- [Just read twice: closing the recall gap for recurrent language models](https://mlsanity.com/paper/2407.05483/) (2024)
- [Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance](https://mlsanity.com/paper/2507.22448/) (2025)
- [Linearizing Large Language Models](https://mlsanity.com/paper/2405.06640/) (2024)
- [GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling](https://mlsanity.com/paper/2311.01927/) (2023)
- [Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search](https://mlsanity.com/paper/2508.15884/) (2025)
- [A Systematic Analysis of Hybrid Linear Attention](https://mlsanity.com/paper/2507.06457/) (2025)
- [ABC: Attention with Bounded-memory Control](https://mlsanity.com/paper/2110.02488/) (2021)
- [Liger: Linearizing Large Language Models to Gated Recurrent Structures](https://mlsanity.com/paper/2503.01496/) (2025)
- [B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memory](https://mlsanity.com/paper/2407.06324/) (2024)
- [RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale](https://mlsanity.com/paper/2505.03005/) (2025)

## Built on by (most cited first, 20 of 20)

- [Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts](https://mlsanity.com/paper/2601.22156/) (2026)
- [LT2: Linear-Time Looped Transformers](https://mlsanity.com/paper/2605.20670/) (2026)
- [MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling](https://mlsanity.com/paper/2602.11761/) (2026)
- [Priming: Hybrid State Space Models From Pre-trained Transformers](https://mlsanity.com/paper/2605.08301/) (2026)
- [Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion](https://mlsanity.com/paper/2605.16579/) (2026)
- [HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization](https://mlsanity.com/paper/2606.20097/) (2026)
- [TACTICL: Task-Aware Compression of Tabular ICL Models](https://mlsanity.com/paper/2608.10837/) (2026)
- [SpikingBrain2.0: Brain-Inspired Foundation Models for Efficient Long-Context and Cross-Platform Inference](https://mlsanity.com/paper/2604.22575/) (2026)
- [Morphing into Hybrid Attention Models](https://mlsanity.com/paper/2606.30562/) (2026)
- [When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models](https://mlsanity.com/paper/2603.26556/) (2026)
- [Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation](https://mlsanity.com/paper/2606.16429/) (2026)
- [NLL-Guided Full-Attention Layer Selection for Training-Free Sliding-Window Adaptation](https://mlsanity.com/paper/2606.27791/) (2026)
- [Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling](https://mlsanity.com/paper/2604.24715/) (2026)
- [DASH: Fast Differentiable Architecture Search for Hybrid Attention in Minutes on a Single GPU](https://mlsanity.com/paper/2605.20936/) (2026)
- [Attention Amnesia in Hybrid LLMs: When CoT Fine-Tuning Breaks Long-Range Recall, and How to Fix It](https://mlsanity.com/paper/2606.11052/) (2026)
- [TinyCeNN-LM: Quality-Gated Conversion of Pretrained Attention with CeNN-Inspired Cellular-Recurrent Layers](https://mlsanity.com/paper/2609.21139/) (2026)
- [On-Policy Attention Linearization](https://mlsanity.com/paper/2609.31947/) (2026)
- [Multilinguality in Hybrid Attention LLMs](https://mlsanity.com/paper/2609.35378/) (2026)
- [The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trends](https://mlsanity.com/paper/2609.39661/) (2026)
- [Tabular Foundation Models Are Effectively Shallow](https://mlsanity.com/paper/2610.13253/) (2026)

## Abstract (arXiv)

Distilling pretrained softmax attention Transformers into more efficient hybrid architectures that interleave softmax and linear attention layers is a promising approach for improving the inference efficiency of LLMs without requiring expensive pretraining from scratch. A critical factor in the conversion process is layer selection, i.e., deciding on which layers to convert to linear attention variants. This paper describes a simple and efficient recipe for layer selection that uses layer importance scores derived from a small amount of training on generic text data. Once the layers have been selected we use a recent pipeline for the distillation process itself citep[RADLADS;][]goldstein2025radlads, which consists of attention weight transfer, hidden state alignment, KL-based distribution matching, followed by a small amount of finetuning. We find that this approach is more effective than existing approaches for layer selection, including heuristics that uniformly interleave linear attentions based on a fixed ratio, as well as more involved approaches that rely on specialized diagnostic datasets.