mlsanityData to 7 Oct 2026

Mask Attention Networks: Rethinking and Strengthen Transformer

arXiv 2103.13597 ↗First posted 25 Mar 2021cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

Transformer is an attention-based neural network, which consists of two sublayers, namely, Self-Attention Network (SAN) and Feed-Forward Network (FFN). Existing research explores to enhance the two sublayers separately to improve the capability of Transformer for text representation. In this paper, we present a novel understanding of SAN and FFN as Mask Attention Networks (MANs) and show that they are two special cases of MANs with static mask matrices. However, their static mask matrices limit the capability for localness modeling in text representation learning. We therefore introduce a new layer named dynamic mask attention network (DMAN) with a learnable mask matrix which is able to model localness adaptively. To incorporate advantages of DMAN, SAN, and FFN, we propose a sequential layered structure to combine the three types of layers. Extensive experiments on various tasks, including neural machine translation and text summarization demonstrate that our model outperforms the original Transformer.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 21 of its 27 references that are in the map. First posted on arXiv 25 Mar 2021.

Builds on

21 of 27 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  3. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  4. 2018Self-Attention with Relative Position Representationscited by 1,112
  5. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  6. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
  7. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  8. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  9. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  10. 2018Scaling Neural Machine Translationcited by 204
  11. 2019Adaptive Attention Span in Transformerscited by 148
  12. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 108
9 more
  1. 2019Augmenting Self-attention with Persistent Memorycited by 92
  2. 2019Monotonic Multihead Attentioncited by 91
  3. 2018Modeling Localness for Self-Attention Networkscited by 43
  4. 2019Pretraining-Based Natural Language Generation for Text Summarizationcited by 40
  5. 2019Convolutional Self-Attention Networkscited by 32
  6. 2019Modeling Recurrence for Transformercited by 26
  7. 2019Context-Aware Self-Attention Networkscited by 25
  8. 2019MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learningcited by 19
  9. 2017Weighted Transformer Network for Machine Translationcited by 16

Built on by

Cited by year

Citing papers per year: 2021: 5; 2022: 7; 2023: 7; 2024: 6; 2025: 3.

28 papers.

Sort:

By area:

  1. 2021A Survey of Transformerscited by 399
  2. 2022Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generationcited by 90
  3. 2024A Temporal Kolmogorov-Arnold Transformer for Time Series Forecastingcited by 50
  4. 2021ProTo: Program-Guided Transformer for Program-Guided Taskscited by 19
  5. 2022Learning Multiscale Transformer Models for Sequence Generationcited by 7
  6. 2023BranchNorm: Robustly Scaling Extremely Deep Transformerscited by 6
  7. 2021Zero-Shot Controlled Generation with Encoder-Decoder Transformerscited by 5
  8. 2022Rethinking Attention Mechanism in Time Series Classificationcited by 5
  9. 2024Theoretical Understanding of In-Context Learning in Shallow Transformers with Unstructured Datacited by 5
  10. 2023On Synthetic Data for Back Translationcited by 4
  11. 2024Multi-layer Learnable Attention Mask for Multimodal Taskscited by 4
  12. 2024Distilled Transformers with Locally Enhanced Global Representations for Face Forgery Detectioncited by 4
16 more
  1. 2021RockGPT: Reconstructing three-dimensional digital rocks from single two-dimensional slice from the perspective of video generationcited by 3
  2. 2022N-gram Is Back: Residual Learning of Neural Text Generation with n-gram Language Modelcited by 3
  3. 2022ANNA: Enhanced Language Representation for Question Answeringcited by 2
  4. 2022FL-Tuning: Layer Tuning for Feed-Forward Network in Transformercited by 2
  5. 2025Gumbel Reranking: Differentiable End-to-End Reranker Optimizationcited by 2
  6. 2025A Semi-amortized Lifted Learning-to-Optimize Masked (SALLO-M) Transformer Model for Scalable and Generalizable Beamformingcited by 2
  7. 2022EIT: Enhanced Interactive Transformercited by 1
  8. 2023InterFormer: Interactive Local and Global Features Fusion for Automatic Speech Recognitioncited by 1
  9. 2023Toward a Deeper Understanding: RetNet Viewed through Convolutioncited by 1
  10. 2023PartialFormer: Modeling Part Instead of Whole for Machine Translationcited by 1
  11. 2023TLM: Token-Level Masking for Transformerscited by 1
  12. 2021Federated Learning for Cross-block Oil-water Layer Identificationcited by 0
  13. 2023TSNet-SAC: Leveraging Transformers for Efficient Task Schedulingcited by 0
  14. 2024Unveiling Vulnerability of Self-Attentioncited by 0
  15. 2024Is Smoothness the Key to Robustness? A Comparison of Attention and Convolution Models Using a Novel Metriccited by 0
  16. 2025MoDA: Modulation Adapter for Fine-Grained Visual Grounding in Instructional MLLMscited by 0