# A Systematic Analysis of Hybrid Linear Attention

arXiv: https://arxiv.org/abs/2507.06457 · HTML: https://mlsanity.com/paper/2507.06457/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 49 references that are in the map.
- First posted on arXiv 8 Jul 2025.

## Builds on (most cited first, 30 of 35)

- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [Retentive Network: A Successor to Transformer for Large Language Models](https://mlsanity.com/paper/2307.08621/) (2023)
- [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://mlsanity.com/paper/2412.06464/) (2024)
- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [Hungry Hungry Hippos: Towards Language Modeling with State Space Models](https://mlsanity.com/paper/2212.14052/) (2022)
- [Parallelizing Linear Transformers with the Delta Rule over Sequence Length](https://mlsanity.com/paper/2406.06484/) (2024)
- [Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models](https://mlsanity.com/paper/2402.19427/) (2024)
- [Repeat After Me: Transformers are Better than State Space Models at Copying](https://mlsanity.com/paper/2402.01032/) (2024)
- [An Empirical Study of Mamba-based Language Models](https://mlsanity.com/paper/2406.07887/) (2024)
- [Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling](https://mlsanity.com/paper/2406.07522/) (2024)
- [MiniMax-01: Scaling Foundation Models with Lightning Attention](https://mlsanity.com/paper/2501.08313/) (2025)
- [You Only Cache Once: Decoder-Decoder Architectures for Language Models](https://mlsanity.com/paper/2405.05254/) (2024)
- [HGRN2: Gated Linear RNNs with State Expansion](https://mlsanity.com/paper/2404.07904/) (2024)
- [Zamba: A Compact 7B SSM Hybrid Model](https://mlsanity.com/paper/2405.16712/) (2024)
- [Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks](https://mlsanity.com/paper/2402.04248/) (2024)
- [RecurrentGemma: Moving Past Transformers for Efficient Open Language Models](https://mlsanity.com/paper/2404.07839/) (2024)
- [Mechanistic Design and Scaling of Hybrid Architectures](https://mlsanity.com/paper/2403.17844/) (2024)
- [Jamba-1.5: Hybrid Transformer-Mamba Models at Scale](https://mlsanity.com/paper/2408.12570/) (2024)
- [Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks](https://mlsanity.com/paper/2405.15731/) (2024)

## Built on by (most cited first, 30 of 39)

- [Distilling to Hybrid Attention Models via KL-Guided Layer Selection](https://mlsanity.com/paper/2512.20569/) (2025)
- [Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts](https://mlsanity.com/paper/2601.22156/) (2026)
- [Hybrid Architectures for Language Models: Systematic Analysis and Design Insights](https://mlsanity.com/paper/2510.04800/) (2025)
- [Scaling Linear Attention with Sparse State Expansion](https://mlsanity.com/paper/2507.16577/) (2025)
- [Short window attention enables long-term memorization](https://mlsanity.com/paper/2509.24552/) (2025)
- [DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone](https://mlsanity.com/paper/2511.15927/) (2025)
- [Rethinking the Role of Efficient Attention in Hybrid Architectures](https://mlsanity.com/paper/2606.15378/) (2026)
- [Component Ablation for Efficient Hybrid Language Model Architectures: Performance, Resilience, and Compression Implications](https://mlsanity.com/paper/2603.22473/) (2026)
- [MDN: Parallelizing Stepwise Momentum for Delta Linear Attention](https://mlsanity.com/paper/2605.05838/) (2026)
- [Memory for Large Language Models](https://mlsanity.com/paper/2607.25380/) (2026)
- [Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction](https://mlsanity.com/paper/2510.20787/) (2025)
- [Controllably Efficient Language Models](https://mlsanity.com/paper/2511.05313/) (2025)
- [Distill-then-Replace: Efficient Task-Specific Hybrid Attention Model Construction](https://mlsanity.com/paper/2601.11667/) (2026)
- [When to Think Fast and Slow? AMOR: Adaptive Entropy Gate for Hybrid Models](https://mlsanity.com/paper/2602.13215/) (2026)
- [Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design](https://mlsanity.com/paper/2609.02986/) (2026)
- [Apriel-H1: Towards Efficient Enterprise Reasoning Models](https://mlsanity.com/paper/2511.02651/) (2025)
- [S0 Tuning: Zero-Overhead Adaptation of Hybrid Recurrent-Attention Models](https://mlsanity.com/paper/2604.01168/) (2026)
- [Sonata: A Hybrid World Model for Inertial Kinematics under Clinical Data Scarcity](https://mlsanity.com/paper/2604.18058/) (2026)
- [HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compression](https://mlsanity.com/paper/2606.28831/) (2026)
- [Morphing into Hybrid Attention Models](https://mlsanity.com/paper/2606.30562/) (2026)
- [Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus](https://mlsanity.com/paper/2608.12149/) (2026)
- [Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale](https://mlsanity.com/paper/2609.11998/) (2026)
- [Sliding Window Recurrences for Sequence Models](https://mlsanity.com/paper/2512.13921/) (2025)
- [Online Vector Quantized Attention](https://mlsanity.com/paper/2602.03922/) (2026)
- [Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation](https://mlsanity.com/paper/2606.16429/) (2026)
- [Where Should LoRA Go? Component-Type Placement in Hybrid Language Models](https://mlsanity.com/paper/2604.22127/) (2026)
- [Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling](https://mlsanity.com/paper/2604.24715/) (2026)
- [DASH: Fast Differentiable Architecture Search for Hybrid Attention in Minutes on a Single GPU](https://mlsanity.com/paper/2605.20936/) (2026)
- [Trading Complexity for Expressivity Through Structured Generalized Linear Token Mixing](https://mlsanity.com/paper/2605.31367/) (2026)
- [Attention Amnesia in Hybrid LLMs: When CoT Fine-Tuning Breaks Long-Range Recall, and How to Fix It](https://mlsanity.com/paper/2606.11052/) (2026)

## Abstract (arXiv)

Transformers face quadratic complexity and memory issues with long sequences, prompting the adoption of linear attention mechanisms using fixed-size hidden states. However, linear models often suffer from limited recall performance, leading to hybrid architectures that combine linear and full attention layers. Despite extensive hybrid architecture research, the choice of linear attention component has not been deeply explored. We systematically evaluate various linear attention models across generations - vector recurrences to advanced gating mechanisms - both standalone and hybridized. To enable this comprehensive analysis, we trained and open-sourced 72 models: 36 at 340M parameters (20B tokens) and 36 at 1.3B parameters (100B tokens), covering six linear attention variants across five hybridization ratios. Benchmarking on standard language modeling and recall tasks reveals that superior standalone linear models do not necessarily excel in hybrids. While language modeling remains stable across linear-to-full attention ratios, recall significantly improves with increased full attention layers, particularly below a 3:1 ratio. Our study highlights selective gating, hierarchical recurrence, and controlled forgetting as critical for effective hybrid models. We recommend architectures such as HGRN-2 or GatedDeltaNet with a linear-to-full ratio between 3:1 and 6:1 to achieve Transformer-level recall efficiently. Our models are open-sourced at https://huggingface.co/collections/m-a-p/hybrid-linear-attention-research-686c488a63d609d2f20e2b1e.