mlsanityData to 14 Oct 2026

HGRN2: Gated Linear RNNs with State Expansion

arXiv 2404.07904 ↗First posted 11 Apr 2024cs.CLcited by 127

Not on the map. Why?

Abstract (arXiv)

Hierarchically gated linear RNN (HGRN, citealtHGRN) has demonstrated competitive training speed and performance in language modeling while offering efficient inference. However, the recurrent state size of HGRN remains relatively small, limiting its expressiveness. To address this issue, we introduce a simple outer product-based state expansion mechanism, which significantly enlarges the recurrent state size without introducing any additional parameters. This enhancement also provides a linear attention interpretation for HGRN2, enabling hardware-efficient training. Our extensive experiments verify the advantage of HGRN2 over HGRN consistently across different settings and competitive with other recurrent models.

Key facts. Cited by 127 papers in the map (data through 14 Oct 2026). Builds on 53 of its 71 references that are in the map. First posted on arXiv 11 Apr 2024.

Builds on

53 of 71 references.

References · 53

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  11. 2020Rethinking Attention with Performerscited by 1,601
  12. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
38 more
  1. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  2. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  3. 2014Neural Turing Machinescited by 644
  4. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  5. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  6. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  7. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  8. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
  9. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  10. 2021Pay Attention to MLPscited by 368
  11. 2021Random Feature Attentioncited by 354
  12. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  13. 2022Long Range Language Modeling via Gated State Spacescited by 265
  14. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
  15. 2022cosFormer: Rethinking Softmax in Attentioncited by 213
  16. 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 208
  17. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 205
  18. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 204
  19. 2022Transformer Quality in Linear Timecited by 191
  20. 2022Mega: Moving Average Equipped Gated Attentioncited by 179
  21. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  22. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 159
  23. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 127
  24. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 102
  25. 2018Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networkscited by 101
  26. 2022The Devil in Linear Transformercited by 98
  27. 2024The Hidden Attention of Mamba Modelscited by 91
  28. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 87
  29. 2024Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Modelscited by 57
  30. 2022Hydra Attention: Efficient Attention with Many Headscited by 55
  31. 2022Pretraining Without Attentioncited by 51
  32. 2023GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modelingcited by 46
  33. 2023Toeplitz Neural Network for Sequence Modelingcited by 43
  34. 2021ABC: Attention with Bounded-memory Controlcited by 36
  35. 2018The unreasonable effectiveness of the forget gatecited by 34
  36. 2022Fine-Tuning Pre-trained Transformers into Decaying Fast Weightscited by 26
  37. 2022Simplifying and Understanding State Space Models with Diagonal Linear RNNscited by 23
  38. 2024State Space Models as Foundation Models: A Control Theoretic Overviewcited by 19

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 23; 2025: 55; 2026: 49.

127 papers.

Sort:

By area:

  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  2. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  3. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  4. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  5. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 174
  6. 2026Kimi K3: Open Frontier Intelligencecited by 138
  7. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 132
  8. 2025A Survey on Latent Reasoningcited by 81
  9. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 74
  10. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Modelscited by 67
  11. 2026Attention Residualscited by 65
  12. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 64
38 more
  1. 2024Longhorn: State Space Models are Amortized Online Learnerscited by 59
  2. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 53
  3. 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 47
  4. 2024The Expressive Capacity of State Space Models: A Formal Language Perspectivecited by 46
  5. 2024LinFusion: 1 GPU, 1 Minute, 16K Imagecited by 43
  6. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 42
  7. 2025Log-Linear Attentioncited by 40
  8. 2025A Systematic Analysis of Hybrid Linear Attentioncited by 39
  9. 2025Liger: Linearizing Large Language Models to Gated Recurrent Structurescited by 33
  10. 2026Gated DeltaNet-2: Decoupling Erase and Write in Linear Attentioncited by 33
  11. 2024Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networkscited by 26
  12. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 22
  13. 2024MetaLA: Unified Optimal Linear Approximation to Softmax Attention Mapcited by 21
  14. 2025Autoregressive Image Generation with Randomized Parallel Decodingcited by 20
  15. 2025Distilling to Hybrid Attention Models via KL-Guided Layer Selectioncited by 20
  16. 2024Scaling Laws for Linear Complexity Language Modelscited by 19
  17. 2025Lattice: Learning to Efficiently Compress the Memorycited by 19
  18. 2024Mamba or RWKV: Exploring High-Quality and High-Efficiency Segment Anything Modelcited by 18
  19. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothingcited by 18
  20. 2025Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Modelscited by 18
  21. 2025Neural Architecture Discovery via Autonomous Evolutioncited by 16
  22. 2025Mamba Modulation: On the Length Generalization of Mambacited by 16
  23. 2025Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanismcited by 15
  24. 2025Comba: Improving Bilinear RNNs with Closed-loop Controlcited by 15
  25. 2024ViG: Linear-complexity Visual Sequence Learning with Gated Linear Attentioncited by 13
  26. 2025Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernelscited by 13
  27. 2026Test-Time Training with KV Binding Is Secretly Linear Attentioncited by 13
  28. 2025Gating is Weighting: Understanding Gated Linear Attention through In-context Learningcited by 12
  29. 2025LoLA: Low-Rank Linear Attention With Sparse Cachingcited by 12
  30. 2024Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulationcited by 11
  31. 2024State Space Models are Strong Text Rerankerscited by 10
  32. 2025LiT: Delving into a Simple Linear Diffusion Transformer for Image Generationcited by 10
  33. 2025GIViC: Generative Implicit Video Compressioncited by 10
  34. 2025Native Hybrid Attention for Efficient Sequence Modelingcited by 10
  35. 2024You Only Scan Once: Efficient Multi-dimension Sequential Modeling with LightNetcited by 9
  36. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mambacited by 9
  37. 2025Hadamard product in deep learning: Introduction, Advances and Challengescited by 9
  38. 2025Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Accesscited by 9

50 most cited of 127 All 127 (JSON)