# Not All Attention Is Needed: Gated Attention Network for Sequence Data

arXiv: https://arxiv.org/abs/1912.00349 · HTML: https://mlsanity.com/paper/1912.00349/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Dec 2019.

## Built on by (most cited first, 13 of 13)

- [Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free](https://mlsanity.com/paper/2505.06708/) (2025)
- [GCA-Net : Utilizing Gated Context Attention for Improving Image Forgery Localization and Detection](https://mlsanity.com/paper/2112.04298/) (2021)
- [Embodied CoT Distillation From LLM To Off-the-shelf Agents](https://mlsanity.com/paper/2412.11499/) (2024)
- [Generating Diversified Comments via Reader-Aware Topic Modeling and Saliency Detection](https://mlsanity.com/paper/2102.06856/) (2021)
- [Attentive Contextual Carryover for Multi-Turn End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2112.06743/) (2021)
- [Cross-Attention is Not Always Needed: Dynamic Cross-Attention for Audio-Visual Dimensional Emotion Recognition](https://mlsanity.com/paper/2403.19554/) (2024)
- [Graph-to-Text Generation with Dynamic Structure Pruning](https://mlsanity.com/paper/2209.07258/) (2022)
- [Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis](https://mlsanity.com/paper/2506.22516/) (2025)
- [Successive Refinement in Large-Scale Computation: Advancing Model Inference Applications](https://mlsanity.com/paper/2402.07229/) (2024)
- [On the Generalizability of Iterative Patch Selection for Memory-Efficient High-Resolution Image Classification](https://mlsanity.com/paper/2412.11237/) (2024)
- [Mamba2 Meets Silence: Robust Vocal Source Separation for Sparse Regions](https://mlsanity.com/paper/2508.14556/) (2025)
- [H-CNN-ViT: A Hierarchical Gated Attention Multi-Branch Model for Bladder Cancer Recurrence Prediction](https://mlsanity.com/paper/2511.13869/) (2025)
- [Discovering Entity-Conditioned Lag Heterogeneity: A Lag-Gated Neural Audit Framework for Panel Time Series](https://mlsanity.com/paper/2605.21542/) (2026)

## Abstract (arXiv)

Although deep neural networks generally have fixed network structures, the concept of dynamic mechanism has drawn more and more attention in recent years. Attention mechanisms compute input-dependent dynamic attention weights for aggregating a sequence of hidden states. Dynamic network configuration in convolutional neural networks (CNNs) selectively activates only part of the network at a time for different inputs. In this paper, we combine the two dynamic mechanisms for text classification tasks. Traditional attention mechanisms attend to the whole sequence of hidden states for an input sentence, while in most cases not all attention is needed especially for long sequences. We propose a novel method called Gated Attention Network (GA-Net) to dynamically select a subset of elements to attend to using an auxiliary network, and compute attention weights to aggregate the selected elements. It avoids a significant amount of unnecessary computation on unattended elements, and allows the model to pay attention to important parts of the sequence. Experiments in various datasets show that the proposed method achieves better performance compared with all baseline models with global or local attention while requiring less computation and achieving better interpretability. It is also promising to extend the idea to more complex attention-based models, such as transformers and seq-to-seq models.