mlsanityData to 7 Oct 2026

Hierarchically Gated Recurrent Neural Network for Sequence Modeling

arXiv 2311.04823 ↗First posted 8 Nov 2023cs.CL cs.LGcited by 100

Not on the map. Why?

Abstract (arXiv)

Transformers have surpassed RNNs in popularity due to their superior abilities in parallel training and long-term dependency modeling. Recently, there has been a renewed interest in using linear RNNs for efficient sequence modeling. These linear RNNs often employ gating mechanisms in the output of the linear recurrence layer while ignoring the significance of using forget gates within the recurrence. In this paper, we propose a gated linear RNN model dubbed Hierarchically Gated Recurrent Neural Network (HGRN), which includes forget gates that are lower bounded by a learnable value. The lower bound increases monotonically when moving up layers. This allows the upper layers to model long-term dependencies and the lower layers to model more local, short-term dependencies. Experiments on language modeling, image classification, and long-range arena benchmarks showcase the efficiency and effectiveness of our proposed model. The source code is available at https://github.com/OpenNLPLab/HGRN.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Builds on 67 of its 81 references that are in the map. First posted on arXiv 8 Nov 2023.

Builds on

67 of 81 references.

References · 67

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
38 more
  1. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  2. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  3. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  4. 2020GLU Variants Improve Transformercited by 1,724
  5. 2021ViViT: A Video Vision Transformercited by 1,610
  6. 2020Rethinking Attention with Performerscited by 1,584
  7. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  8. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  9. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  10. 2021AST: Audio Spectrogram Transformercited by 634
  11. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  12. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  13. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  14. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  15. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  16. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  17. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  18. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  19. 2021Pay Attention to MLPscited by 367
  20. 2021Random Feature Attentioncited by 350
  21. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  22. 2015LSTM: A Search Space Odysseycited by 315
  23. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  24. 2022Long Range Language Modeling via Gated State Spacescited by 265
  25. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  26. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  27. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  28. 2022Transformer Quality in Linear Timecited by 189
  29. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  30. 2022Liquid Structural State-Space Modelscited by 134
  31. 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
  32. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 126
  33. 2016Hierarchical Multiscale Recurrent Neural Networkscited by 102
  34. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  35. 2018Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networkscited by 101
  36. 2022The Devil in Linear Transformercited by 98
  37. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 86
  38. 2017Dilated Recurrent Neural Networkscited by 83

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 42; 2025: 35; 2026: 20.

100 papers.

Sort:

By area:

  1. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  2. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  3. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  4. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  5. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  6. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  7. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  8. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  9. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  10. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  11. 2024Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architecturescited by 97
  12. 2023Diffusion Models Without Attentioncited by 75
38 more
  1. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
  2. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  3. 2024Longhorn: State Space Models are Amortized Online Learnerscited by 58
  4. 2024Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Modelscited by 57
  5. 2024DiG: Scalable and Efficient Diffusion Models with Gated Linear Attentioncited by 56
  6. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  7. 2024Base of RoPE Bounds Context Lengthcited by 49
  8. 2024The Expressive Capacity of State Space Models: A Formal Language Perspectivecited by 45
  9. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45
  10. 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 44
  11. 2024Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Modelscited by 41
  12. 2024MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selectioncited by 40
  13. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  14. 2024LION: Linear Group RNN for 3D Object Detection in Point Cloudscited by 38
  15. 2025Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layerscited by 32
  16. 2024Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attentioncited by 31
  17. 2024MambaLRP: Explaining Selective State Space Sequence Modelscited by 31
  18. 2024Scalable MatMul-free Language Modelingcited by 21
  19. 2025Implicit Language Models are RNNs: Balancing Parallelization and Expressivitycited by 21
  20. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 21
  21. 2024MetaLA: Unified Optimal Linear Approximation to Softmax Attention Mapcited by 20
  22. 2024Mamba or RWKV: Exploring High-Quality and High-Efficiency Segment Anything Modelcited by 18
  23. 2024CO2: Efficient Distributed Training with Full Communication-Computation Overlapcited by 17
  24. 2024On the Resurgence of Recurrent Models for Long Sequences -- Survey and Research Opportunities in the Transformer Eracited by 17
  25. 2024ViG: Linear-complexity Visual Sequence Learning with Gated Linear Attentioncited by 13
  26. 2025Mamba Modulation: On the Length Generalization of Mambacited by 13
  27. 2025Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernelscited by 12
  28. 2025Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformerscited by 12
  29. 2025Scaling Linear Attention with Sparse State Expansioncited by 12
  30. 2024Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulationcited by 11
  31. 2023On the Long Range Abilities of Transformerscited by 10
  32. 2024SIGMA: Selective Gated Mamba for Sequential Recommendationcited by 10
  33. 2026Test-Time Training with KV Binding Is Secretly Linear Attentioncited by 10
  34. 2024You Only Scan Once: Efficient Multi-dimension Sequential Modeling with LightNetcited by 9
  35. 2025LiT: Delving into a Simple Linear Diffusion Transformer for Image Generationcited by 9
  36. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mambacited by 9
  37. 2024Linear Attention Sequence Parallelismcited by 8
  38. 2024Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Modelscited by 8

50 most cited of 100 All 100 (JSON)