mlsanityData to 7 Oct 2026

Parallelizing Linear Transformers with the Delta Rule over Sequence Length

arXiv 2406.06484 ↗First posted 10 Jun 2024cs.LG cs.CLcited by 373

Not on the map. Why?

Cited by 33 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformers with linear attention (i.e., linear transformers) and state-space models have recently been suggested as a viable linear-time alternative to transformers with softmax attention. However, these models still underperform transformers especially on tasks that require in-context retrieval. While more expressive variants of linear transformers which replace the additive update in linear transformers with the delta rule (DeltaNet) have been found to be more effective at associative recall, existing algorithms for training such models do not parallelize over sequence length and are thus inefficient to train on modern hardware. This work describes a hardware-efficient algorithm for training linear transformers with the delta rule, which exploits a memory-efficient representation for computing products of Householder matrices. This algorithm allows us to scale up DeltaNet to standard language modeling settings. We train a 1.3B model for 100B tokens and find that it outperforms recent linear-time baselines such as Mamba and GLA in terms of perplexity and zero-shot performance on downstream tasks. We also experiment with two hybrid models which combine DeltaNet layers with (1) sliding-window attention layers every other layer or (2) two global attention layers, and find that these hybrids outperform strong transformer baselines.

Key facts. Cited by 373 papers in the map (data through 7 Oct 2026). Builds on 91 of its 135 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

91 of 135 references.

References · 91

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
38 more
  1. 2020GLU Variants Improve Transformercited by 1,724
  2. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  3. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  4. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  5. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  6. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  7. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  8. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  9. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  10. 2014Neural Turing Machinescited by 630
  11. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  12. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  13. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  14. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  15. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  16. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  17. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  18. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  19. 2021Random Feature Attentioncited by 350
  20. 2024xLSTM: Extended Long Short-Term Memorycited by 320
  21. 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 197
  22. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  23. 2022Transformer Quality in Linear Timecited by 189
  24. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 188
  25. 2024The Illusion of State in State-Space Modelscited by 181
  26. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  27. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  28. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  29. 2024Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spacescited by 132
  30. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 126
  31. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  32. 2020Large Associative Memory Problem in Neurobiology and Machine Learningcited by 110
  33. 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 108
  34. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 104
  35. 2023Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakescited by 101
  36. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  37. 2022The Devil in Linear Transformercited by 98
  38. 2021Finetuning Pretrained Transformers into RNNscited by 94

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 25; 2025: 135; 2026: 213.

373 papers.

Sort:

By area:

  1. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  2. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  3. 2024Titans: Learning to Memorize at Test Timecited by 303
  4. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  5. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  6. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  7. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  8. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  9. 2025TTT3R: 3D Reconstruction as Test-Time Trainingcited by 122
  10. 2025Test-Time Training Done Rightcited by 121
  11. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  12. 2025One-Minute Video Generation with Test-Time Trainingcited by 100
38 more
  1. 2026Kimi K3: Open Frontier Intelligencecited by 99
  2. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 97
  3. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  4. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 76
  5. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  6. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
  7. 2025A Survey on Latent Reasoningcited by 71
  8. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 69
  9. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  10. 2024Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decodingcited by 65
  11. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  12. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
  13. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 60
  14. 2025Long-Context State-Space Video World Modelscited by 59
  15. 2024Longhorn: State Space Models are Amortized Online Learnerscited by 58
  16. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 57
  17. 2025End-to-End Test-Time Training for Long Contextcited by 53
  18. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  19. 2025Baichuan-M1: Pushing the Medical Capability of Large Language Modelscited by 41
  20. 2025Log-Linear Attentioncited by 39
  21. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  22. 2024Towards Scalable and Stable Parallelization of Nonlinear RNNscited by 38
  23. 2025A Systematic Analysis of Hybrid Linear Attentioncited by 38
  24. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 37
  25. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 32
  26. 2025Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layerscited by 32
  27. 2025Understanding Transformer from the Perspective of Associative Memorycited by 31
  28. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 29
  29. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 28
  30. 2025MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connectionscited by 27
  31. 2024LoLCATs: On Low-Rank Linearizing of Large Language Modelscited by 25
  32. 2025The emergence of sparse attention: impact of data distribution and benefits of repetitioncited by 25
  33. 2026Scal3R: Scalable Test-Time Training for Large-Scale 3D Reconstructioncited by 25
  34. 2026Gated DeltaNet-2: Decoupling Erase and Write in Linear Attentioncited by 24
  35. 2026ZipMap: Linear-Time Stateful 3D Reconstruction via Test-Time Trainingcited by 23
  36. 2025Implicit Language Models are RNNs: Balancing Parallelization and Expressivitycited by 21
  37. 2025Sliding Window Attention Training for Efficient Large Language Modelscited by 21
  38. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 21

50 most cited of 373 All 373 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026On-Policy Attention Linearization
  2. 2025DeltaLLM: A Training-Free Framework Exploiting Temporal Sparsity for Efficient Edge LLM Inference
  3. 2026RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State
  4. 2026Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts
  5. 2025How Transformers Learn In-Context Recall Tasks? Optimality, Training Dynamics and Generalization
  6. 2026Retrieval-Aware Distillation for Transformer-SSM Hybrids
  7. 2024On the Power of Convolution Augmented Transformer
  8. 2022Efficient Long Sequence Modeling via State Space Augmented Transformer