mlsanityData to 7 Oct 2026

Gated Delta Networks: Improving Mamba2 with Delta Rule

arXiv 2412.06464 ↗First posted 9 Dec 2024cs.CL cs.LGcited by 483

Not on the map. Why?

Cited by 60 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Linear Transformers have gained attention as efficient alternatives to standard Transformers, but their performance in retrieval and long-context tasks has been limited. To address these limitations, recent work has explored two distinct mechanisms: gating for adaptive memory control and the delta update rule for precise memory modifications. We observe that these mechanisms are complementary: gating enables rapid memory erasure while the delta rule facilitates targeted updates. Building on this insight, we introduce the gated delta rule and develop a parallel training algorithm optimized for modern hardware. Our proposed architecture, Gated DeltaNet, consistently surpasses existing models like Mamba2 and DeltaNet across multiple benchmarks, including language modeling, common-sense reasoning, in-context retrieval, length extrapolation, and long-context understanding. We further enhance performance by developing hybrid architectures that combine Gated DeltaNet layers with sliding window attention or Mamba2 layers, achieving both improved training efficiency and superior task performance.

Key facts. Cited by 483 papers in the map (data through 7 Oct 2026). Builds on 73 of its 100 references that are in the map. First posted on arXiv 9 Dec 2024.

Builds on

73 of 100 references.

References · 73

  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  12. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
38 more
  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  2. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  4. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  5. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  6. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  7. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  8. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  9. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  10. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  11. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  12. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  13. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  14. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  15. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  16. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  17. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  18. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  19. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  20. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  21. 2021Random Feature Attentioncited by 350
  22. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  23. 2021Efficient Attentions for Long Document Summarizationcited by 342
  24. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  25. 2024xLSTM: Extended Long Short-Term Memorycited by 320
  26. 2015LSTM: A Search Space Odysseycited by 315
  27. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  28. 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 197
  29. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  30. 2022Transformer Quality in Linear Timecited by 189
  31. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 188
  32. 2024The Illusion of State in State-Space Modelscited by 181
  33. 2024An Empirical Study of Mamba-based Language Modelscited by 177
  34. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  35. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  36. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 126
  37. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  38. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 125; 2026: 355.

483 papers.

Sort:

By area:

  1. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  2. 2024Titans: Learning to Memorize at Test Timecited by 303
  3. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  4. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  5. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  6. 2025One-Minute Video Generation with Test-Time Trainingcited by 100
  7. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 100
  8. 2026Kimi K3: Open Frontier Intelligencecited by 99
  9. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 76
  10. 2025A Survey on Latent Reasoningcited by 71
  11. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 69
  12. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
38 more
  1. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  2. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
  3. 2025SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Trainingcited by 63
  4. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMscited by 62
  5. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 60
  6. 2025Long-Context State-Space Video World Modelscited by 59
  7. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 57
  8. 2026Attention Residualscited by 57
  9. 2025SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attentioncited by 55
  10. 2025End-to-End Test-Time Training for Long Contextcited by 53
  11. 2026SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformercited by 50
  12. 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 44
  13. 2025Baichuan-M1: Pushing the Medical Capability of Large Language Modelscited by 41
  14. 2025Log-Linear Attentioncited by 39
  15. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  16. 2025VideoSSM: Autoregressive Long Video Generation with Hybrid State-Space Memorycited by 39
  17. 2025A Systematic Analysis of Hybrid Linear Attentioncited by 38
  18. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 37
  19. 2026MiniMax Sparse Attentioncited by 34
  20. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 32
  21. 2025Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layerscited by 32
  22. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 27
  23. 2025Pack and Force Your Memory: Long-form and Consistent Video Generationcited by 26
  24. 2025RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scalecited by 24
  25. 2026Benchmark Test-Time Scaling of General LLM Agentscited by 24
  26. 2026Gated DeltaNet-2: Decoupling Erase and Write in Linear Attentioncited by 24
  27. 2026The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinkscited by 23
  28. 2025Sliding Window Attention Training for Efficient Large Language Modelscited by 21
  29. 2025SageAttention2++: A More Efficient Implementation of SageAttention2cited by 21
  30. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 21
  31. 2026A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Trainingcited by 21
  32. 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATcited by 21
  33. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 20
  34. 2025Understanding and Improving Length Generalization in Recurrent Modelscited by 19
  35. 2025Lattice: Learning to Efficiently Compress the Memorycited by 18
  36. 2025Distilling to Hybrid Attention Models via KL-Guided Layer Selectioncited by 17
  37. 2025Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Modelscited by 16
  38. 2026Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contextscited by 16

50 most cited of 483 All 483 (JSON)