mlsanityData to 7 Oct 2026

Mega: Moving Average Equipped Gated Attention

arXiv 2209.10655 ↗First posted 21 Sep 2022cs.LGcited by 178

Not on the map. Why?

Cited by 27 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The design choices in the Transformer attention mechanism, including weak inductive bias and quadratic computational complexity, have limited its application for modeling long sequences. In this paper, we introduce Mega, a simple, theoretically grounded, single-head gated attention mechanism equipped with (exponential) moving average to incorporate inductive bias of position-aware local dependencies into the position-agnostic attention mechanism. We further propose a variant of Mega that offers linear time and space complexity yet yields only minimal quality loss, by efficiently splitting the whole sequence into multiple chunks with fixed length. Extensive experiments on a wide range of sequence modeling benchmarks, including the Long Range Arena, neural machine translation, auto-regressive language modeling, and image and speech classification, show that Mega achieves significant improvements over other sequence models, including variants of Transformers and recent state space models.

Key facts. Cited by 178 papers in the map (data through 7 Oct 2026). Builds on 62 of its 81 references that are in the map. First posted on arXiv 21 Sep 2022.

Builds on

62 of 81 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  10. 2020Longformer: The Long-Document Transformercited by 3,362
  11. 2016Pointer Sentinel Mixture Modelscited by 2,992
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
38 more
  1. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  2. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  3. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2020Reformer: The Efficient Transformercited by 1,658
  7. 2020Rethinking Attention with Performerscited by 1,584
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  10. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  11. 2017Random Erasing Data Augmentationcited by 1,235
  12. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  13. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  14. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  15. 2016Deep Networks with Stochastic Depthcited by 973
  16. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  17. 2020Efficient Transformers: A Surveycited by 784
  18. 2018Image Transformercited by 770
  19. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  20. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  21. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  22. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  23. 2016Error bounds for approximations with deep ReLU networkscited by 567
  24. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  25. 2017Convolutional Sequence to Sequence Learningcited by 550
  26. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  27. 2021Pay Attention to MLPscited by 367
  28. 2021Random Feature Attentioncited by 350
  29. 2022Long Range Language Modeling via Gated State Spacescited by 265
  30. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  31. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  32. 2018Scaling Neural Machine Translationcited by 204
  33. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  34. 2022Transformer Quality in Linear Timecited by 189
  35. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  36. 2017Searching for Activation Functionscited by 158
  37. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  38. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 128

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 48; 2024: 72; 2025: 35; 2026: 17.

178 papers.

Sort:

By area:

  1. 2024VMamba: Visual State Space Modelcited by 955
  2. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  3. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  4. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  5. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  6. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  7. 2023Adapting Language Models to Compress Contextscited by 350
  8. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  9. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  10. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  11. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  12. 2022A Length-Extrapolatable Transformercited by 123
38 more
  1. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  2. 2025Test-Time Training Done Rightcited by 121
  3. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  4. 2024The Hidden Attention of Mamba Modelscited by 88
  5. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 86
  6. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  7. 2024SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time seriescited by 81
  8. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 77
  9. 2023Diffusion Models Without Attentioncited by 75
  10. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 71
  11. 2024Point Mamba: A Novel Point Cloud Backbone Based on State Space Model with Octree-Based Ordering Strategycited by 71
  12. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  13. 2023Simple Hardware-Efficient Long Convolutions for Sequence Modelingcited by 62
  14. 2024MoE-Mamba: Efficient Selective State Space Models with Mixture of Expertscited by 60
  15. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  16. 2024Mechanistic Design and Scaling of Hybrid Architecturescited by 57
  17. 2023CARD: Channel Aligned Robust Blend Transformer for Time Series Forecastingcited by 55
  18. 2022Pretraining Without Attentioncited by 51
  19. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  20. 2024MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Modelscited by 46
  21. 2024Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Lengthcited by 46
  22. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  23. 2024Scalable Diffusion Models with State Space Backbonecited by 45
  24. 2024A Survey on Transformer Compressioncited by 44
  25. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  26. 2024Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Modelscited by 41
  27. 2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37
  28. 2024VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecastingcited by 35
  29. 2023Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priorscited by 33
  30. 2023FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Corescited by 32
  31. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 32
  32. 2024MambaLRP: Explaining Selective State Space Sequence Modelscited by 31
  33. 2023Transformer-VQ: Linear-Time Transformers via Vector Quantizationcited by 26
  34. 2023Findings of the WMT 2023 Shared Task on Discourse-Level Literary Translation: A Fresh Orb in the Cosmos of LLMscited by 25
  35. 2024Learning Enriched Features via Selective State Spaces Model for Efficient Image Deblurringcited by 25
  36. 2023Block-State Transformerscited by 24
  37. 2024Matten: Video Generation with Mamba-Attentioncited by 24
  38. 2022Simplifying and Understanding State Space Models with Diagonal Linear RNNscited by 23

50 most cited of 178 All 178 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Sliding Window Attention Training for Efficient Large Language Models
  2. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation
  3. 2025Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer
  4. 2023Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation
  5. 2026Blurry Window Attention
  6. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
  7. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
  8. 2025GatedFWA: Linear Flash Windowed Attention with Gated Associative Memory