# Liger: Linearizing Large Language Models to Gated Recurrent Structures

arXiv: https://arxiv.org/abs/2503.01496 · HTML: https://mlsanity.com/paper/2503.01496/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 55 references that are in the map.
- First posted on arXiv 3 Mar 2025.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [Retentive Network: A Successor to Transformer for Large Language Models](https://mlsanity.com/paper/2307.08621/) (2023)
- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)
- [Jamba: A Hybrid Transformer-Mamba Language Model](https://mlsanity.com/paper/2403.19887/) (2024)
- [Titans: Learning to Memorize at Test Time](https://mlsanity.com/paper/2501.00663/) (2024)
- [Simple linear attention language models balance the recall-throughput tradeoff](https://mlsanity.com/paper/2402.18668/) (2024)
- [MiniMax-01: Scaling Foundation Models with Lightning Attention](https://mlsanity.com/paper/2501.08313/) (2025)
- [Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence](https://mlsanity.com/paper/2404.05892/) (2024)
- [You Only Cache Once: Decoder-Decoder Architectures for Language Models](https://mlsanity.com/paper/2405.05254/) (2024)
- [HGRN2: Gated Linear RNNs with State Expansion](https://mlsanity.com/paper/2404.07904/) (2024)
- [Zamba: A Compact 7B SSM Hybrid Model](https://mlsanity.com/paper/2405.16712/) (2024)
- [LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training](https://mlsanity.com/paper/2406.16554/) (2024)
- [The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry](https://mlsanity.com/paper/2402.04347/) (2024)
- [Hymba: A Hybrid-head Architecture for Small Language Models](https://mlsanity.com/paper/2411.13676/) (2024)
- [Finetuning Pretrained Transformers into RNNs](https://mlsanity.com/paper/2103.13076/) (2021)
- [Agent Attention: On the Integration of Softmax and Linear Attention](https://mlsanity.com/paper/2312.08874/) (2023)

## Built on by (most cited first, 30 of 33)

- [Kimi Linear: An Expressive, Efficient Attention Architecture](https://mlsanity.com/paper/2510.26692/) (2025)
- [A Survey on Latent Reasoning](https://mlsanity.com/paper/2507.06203/) (2025)
- [Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search](https://mlsanity.com/paper/2508.15884/) (2025)
- [Distilling to Hybrid Attention Models via KL-Guided Layer Selection](https://mlsanity.com/paper/2512.20569/) (2025)
- [Towards AI Search Paradigm](https://mlsanity.com/paper/2506.17188/) (2025)
- [Comba: Improving Bilinear RNNs with Closed-loop Control](https://mlsanity.com/paper/2506.02475/) (2025)
- [Hybrid Architectures for Language Models: Systematic Analysis and Design Insights](https://mlsanity.com/paper/2510.04800/) (2025)
- [LoLA: Low-Rank Linear Attention With Sparse Caching](https://mlsanity.com/paper/2505.23666/) (2025)
- [KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction](https://mlsanity.com/paper/2510.05373/) (2025)
- [Lizard: An Efficient Linearization Framework for Large Language Models](https://mlsanity.com/paper/2507.09025/) (2025)
- [STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs](https://mlsanity.com/paper/2602.02180/) (2026)
- [ViT-AdaLA: Adapting Vision Transformers with Linear Attention](https://mlsanity.com/paper/2603.16063/) (2026)
- [HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization](https://mlsanity.com/paper/2606.20097/) (2026)
- [Effective Distillation to Hybrid xLSTM Architectures](https://mlsanity.com/paper/2603.15590/) (2026)
- [Solar Open 2 Technical Report](https://mlsanity.com/paper/2607.20062/) (2026)
- [TPTT: Transforming Pretrained Transformers into Titans](https://mlsanity.com/paper/2506.17671/) (2025)
- [Untangling Component Imbalance in Hybrid Linear Attention Conversion Methods](https://mlsanity.com/paper/2510.05901/) (2025)
- [SpikingBrain2.0: Brain-Inspired Foundation Models for Efficient Long-Context and Cross-Platform Inference](https://mlsanity.com/paper/2604.22575/) (2026)
- [Morphing into Hybrid Attention Models](https://mlsanity.com/paper/2606.30562/) (2026)
- [Sliding-window beats linear attention](https://mlsanity.com/paper/2608.28444/) (2026)
- [A State-Transition Framework for Efficient LLM Reasoning](https://mlsanity.com/paper/2602.01198/) (2026)
- [On Subquadratic Architectures: From Applications to Principles](https://mlsanity.com/paper/2606.12364/) (2026)
- [Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation](https://mlsanity.com/paper/2606.16429/) (2026)
- [The Key to Going Linear: Analysis-Driven Transformer Linearization](https://mlsanity.com/paper/2607.07706/) (2026)
- [LayerBoost: Layer-Aware Attention Reduction for Efficient LLMs](https://mlsanity.com/paper/2604.22050/) (2026)
- [Memory by Design: Bayesian Layers for Sequence Modeling](https://mlsanity.com/paper/2605.31163/) (2026)
- [LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning](https://mlsanity.com/paper/2607.19358/) (2026)
- [GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference](https://mlsanity.com/paper/2607.24788/) (2026)
- [On-Policy Attention Linearization](https://mlsanity.com/paper/2609.31947/) (2026)
- [CyFA: Linear Sequence Modeling with Relative-Time-Partitioned Memory](https://mlsanity.com/paper/2609.36259/) (2026)

## Abstract (arXiv)

Transformers with linear recurrent modeling offer linear-time training and constant-memory inference. Despite their demonstrated efficiency and performance, pretraining such non-standard architectures from scratch remains costly and risky. The linearization of large language models (LLMs) transforms pretrained standard models into linear recurrent structures, enabling more efficient deployment. However, current linearization methods typically introduce additional feature map modules that require extensive fine-tuning and overlook the gating mechanisms used in state-of-the-art linear recurrent models. To address these issues, this paper presents Liger, short for Linearizing LLMs to gated recurrent structures. Liger is a novel approach for converting pretrained LLMs into gated linear recurrent models without adding extra parameters. It repurposes the pretrained key matrix weights to construct diverse gating mechanisms, facilitating the formation of various gated recurrent structures while avoiding the need to train additional components from scratch. Using lightweight fine-tuning with Low-Rank Adaptation (LoRA), Liger restores the performance of the linearized gated recurrent models to match that of the original LLMs. Additionally, we introduce Liger Attention, an intra-layer hybrid attention mechanism, which significantly recovers 93% of the Transformer-based LLM at 0.02% pre-training tokens during the linearization process, achieving competitive results across multiple benchmarks, as validated on models ranging from 1B to 8B parameters. Code is available at https://github.com/OpenSparseLLMs/Linearization.