mlsanityData to 7 Oct 2026

Linearizing Large Language Models

arXiv 2405.06640 ↗First posted 10 May 2024cs.CLcited by 51

Not on the map. Why?

Abstract (arXiv)

Linear transformers have emerged as a subquadratic-time alternative to softmax attention and have garnered significant interest due to their fixed-size recurrent state that lowers inference cost. However, their original formulation suffers from poor scaling and underperforms compute-matched transformers. Recent linear models such as RWKV and Mamba have attempted to address these shortcomings by proposing novel time-mixing and gating architectures, but pre-training large language models requires significant data and compute investments. Thus, the search for subquadratic architectures is limited by the availability of compute and quality pre-training datasets. As a cost-effective alternative to pre-training linear transformers, we propose Scalable UPtraining for Recurrent Attention (SUPRA). We present a method to uptrain existing large pre-trained transformers into Recurrent Neural Networks (RNNs) with a modest compute budget. This allows us to leverage the strong pre-training data and performance of existing transformer LLMs, while requiring 5% of the training cost. We find that our linearization technique leads to competitive performance on standard benchmarks, but we identify persistent in-context learning and long-context modeling shortfalls for even the largest linear models. Our code and models can be found at https://github.com/TRI-ML/linear_open_lm.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 35 of its 42 references that are in the map. First posted on arXiv 10 May 2024.

Builds on

35 of 42 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2016Layer Normalizationcited by 5,579
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2023Mistral 7Bcited by 3,065
  9. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  11. 2020Rethinking Attention with Performerscited by 1,584
  12. 2018Group Normalizationcited by 1,514
23 more
  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  2. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  3. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  4. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  5. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  6. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  7. 2022Efficiently Scaling Transformer Inferencecited by 529
  8. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  9. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  10. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  11. 2021Random Feature Attentioncited by 350
  12. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  13. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  14. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  15. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  16. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 119
  17. 2021An Attention Free Transformercited by 111
  18. 2022The Devil in Linear Transformercited by 98
  19. 2021Luna: Linear Unified Nested Attentioncited by 96
  20. 2021Finetuning Pretrained Transformers into RNNscited by 94
  21. 2024In-Context Language Learning: Architectures and Algorithmscited by 86
  22. 2024Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Modelscited by 57
  23. 2020Higher Order Linear Transformercited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 25; 2026: 15.

51 papers.

Sort:

By area:

  1. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  2. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  3. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  4. 2025A Survey on Latent Reasoningcited by 71
  5. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Modelscited by 66
  6. 2024Falcon Mamba: The First Competitive Attention-free 7B Language Modelcited by 43
  7. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrencecited by 41
  8. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  9. 2025Liger: Linearizing Large Language Models to Gated Recurrent Structurescited by 32
  10. 2024LoLCATs: On Low-Rank Linearizing of Large Language Modelscited by 25
  11. 2025RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scalecited by 24
  12. 2024Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Timecited by 22
38 more
  1. 2025Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasonerscited by 22
  2. 2025M1: Towards Scalable Test-Time Compute with Mamba Reasoning Modelscited by 21
  3. 2024A Tighter Complexity Analysis of SparseGPTcited by 19
  4. 2024HSR-Enhanced Sparse Attention Accelerationcited by 19
  5. 2025Distilling to Hybrid Attention Models via KL-Guided Layer Selectioncited by 17
  6. 2026VGG-T3: Offline Feed-Forward 3D Reconstruction at Scalecited by 14
  7. 2025Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformercited by 13
  8. 2024LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptationcited by 11
  9. 2025Multimodal Mamba: Decoder-only Multimodal State Space Model via Quadratic to Linear Distillationcited by 11
  10. 2025LoLA: Low-Rank Linear Attention With Sparse Cachingcited by 10
  11. 2024QUART-Online: Latency-Free Large Multimodal Language Model for Quadruped Robot Learningcited by 9
  12. 2025Lizard: An Efficient Linearization Framework for Large Language Modelscited by 6
  13. 2025ViT-Linearizer: Distilling Quadratic Knowledge into Linear-Time Vision Modelscited by 5
  14. 2025SpikingBrain: Spiking Brain-inspired Large Modelscited by 5
  15. 2025PrefixMemory-Tuning: Modernizing Prefix-Tuning by Decoupling the Prefix from Attentioncited by 4
  16. 2025On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attentioncited by 3
  17. 2025Diffusion Transformer-to-Mamba Distillation for High-Resolution Image Generationcited by 3
  18. 2025LAWCAT: Efficient Distillation from Quadratic to Linear Attention with Convolution across Tokens for Long Context Modelingcited by 3
  19. 2026STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMscited by 3
  20. 2026Effective Distillation to Hybrid xLSTM Architecturescited by 3
  21. 2026Attention to Mamba: A Recipe for Cross-Architecture Distillationcited by 3
  22. 2025Theoretical Foundation of Flow-Based Time Series Generation: Provable Approximation, Generalization, and Efficiencycited by 2
  23. 2025Untangling Component Imbalance in Hybrid Linear Attention Conversion Methodscited by 2
  24. 2026SpikingBrain2.0: Brain-Inspired Foundation Models for Efficient Long-Context and Cross-Platform Inferencecited by 2
  25. 2026Morphing into Hybrid Attention Modelscited by 2
  26. 2026Solar Open 2 Technical Reportcited by 2
  27. 2026Sliding-window beats linear attentioncited by 2
  28. 2024Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexitycited by 1
  29. 2025Efficient Large Language Model Inference with Neural Block Linearizationcited by 1
  30. 2025TPTT: Transforming Pretrained Transformers into Titanscited by 1
  31. 2026When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Modelscited by 1
  32. 2026On Subquadratic Architectures: From Applications to Principlescited by 1
  33. 2026Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillationcited by 1
  34. 2026The Key to Going Linear: Analysis-Driven Transformer Linearizationcited by 1
  35. 2025ChunkLLM: A Lightweight Pluggable Framework for Accelerating LLMs Inferencecited by 0
  36. 2025Network of Theseus (like the ship)cited by 0
  37. 2026LayerBoost: Layer-Aware Attention Reduction for Efficient LLMscited by 0
  38. 2026On-Policy Attention Linearizationcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models
  2. 2026Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts
  3. 2025Sliding Window Attention Training for Efficient Large Language Models
  4. 2024RecurFormer: Not All Transformer Heads Need Self-Attention
  5. 2026M2RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling
  6. 2024Linear Transformers with Learnable Kernel Functions are Better In-Context Models
  7. 2024Transformer Layer Injection: A Novel Approach for Efficient Upscaling of Large Language Models
  8. 2024Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix