# Mask Attention Networks: Rethinking and Strengthen Transformer

arXiv: https://arxiv.org/abs/2103.13597 · HTML: https://mlsanity.com/paper/2103.13597/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 25 Mar 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [A Neural Attention Model for Abstractive Sentence Summarization](https://mlsanity.com/paper/1509.00685/) (2015)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [Scaling Neural Machine Translation](https://mlsanity.com/paper/1806.00187/) (2018)
- [Adaptive Attention Span in Transformers](https://mlsanity.com/paper/1905.07799/) (2019)
- [Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View](https://mlsanity.com/paper/1906.02762/) (2019)
- [Augmenting Self-attention with Persistent Memory](https://mlsanity.com/paper/1907.01470/) (2019)
- [Monotonic Multihead Attention](https://mlsanity.com/paper/1909.12406/) (2019)
- [Modeling Localness for Self-Attention Networks](https://mlsanity.com/paper/1810.10182/) (2018)
- [Pretraining-Based Natural Language Generation for Text Summarization](https://mlsanity.com/paper/1902.09243/) (2019)
- [Convolutional Self-Attention Networks](https://mlsanity.com/paper/1904.03107/) (2019)
- [Modeling Recurrence for Transformer](https://mlsanity.com/paper/1904.03092/) (2019)
- [Context-Aware Self-Attention Networks](https://mlsanity.com/paper/1902.05766/) (2019)
- [MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning](https://mlsanity.com/paper/1911.09483/) (2019)
- [Weighted Transformer Network for Machine Translation](https://mlsanity.com/paper/1711.02132/) (2017)

## Built on by (most cited first, 28 of 28)

- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation](https://mlsanity.com/paper/2206.02369/) (2022)
- [A Temporal Kolmogorov-Arnold Transformer for Time Series Forecasting](https://mlsanity.com/paper/2406.02486/) (2024)
- [ProTo: Program-Guided Transformer for Program-Guided Tasks](https://mlsanity.com/paper/2110.00804/) (2021)
- [Learning Multiscale Transformer Models for Sequence Generation](https://mlsanity.com/paper/2206.09337/) (2022)
- [BranchNorm: Robustly Scaling Extremely Deep Transformers](https://mlsanity.com/paper/2305.02790/) (2023)
- [Zero-Shot Controlled Generation with Encoder-Decoder Transformers](https://mlsanity.com/paper/2106.06411/) (2021)
- [Rethinking Attention Mechanism in Time Series Classification](https://mlsanity.com/paper/2207.07564/) (2022)
- [Theoretical Understanding of In-Context Learning in Shallow Transformers with Unstructured Data](https://mlsanity.com/paper/2402.00743/) (2024)
- [On Synthetic Data for Back Translation](https://mlsanity.com/paper/2310.13675/) (2023)
- [Multi-layer Learnable Attention Mask for Multimodal Tasks](https://mlsanity.com/paper/2406.02761/) (2024)
- [Distilled Transformers with Locally Enhanced Global Representations for Face Forgery Detection](https://mlsanity.com/paper/2412.20156/) (2024)
- [RockGPT: Reconstructing three-dimensional digital rocks from single two-dimensional slice from the perspective of video generation](https://mlsanity.com/paper/2108.03132/) (2021)
- [N-gram Is Back: Residual Learning of Neural Text Generation with n-gram Language Model](https://mlsanity.com/paper/2210.14431/) (2022)
- [ANNA: Enhanced Language Representation for Question Answering](https://mlsanity.com/paper/2203.14507/) (2022)
- [FL-Tuning: Layer Tuning for Feed-Forward Network in Transformer](https://mlsanity.com/paper/2206.15312/) (2022)
- [Gumbel Reranking: Differentiable End-to-End Reranker Optimization](https://mlsanity.com/paper/2502.11116/) (2025)
- [A Semi-amortized Lifted Learning-to-Optimize Masked (SALLO-M) Transformer Model for Scalable and Generalizable Beamforming](https://mlsanity.com/paper/2510.13077/) (2025)
- [EIT: Enhanced Interactive Transformer](https://mlsanity.com/paper/2212.10197/) (2022)
- [InterFormer: Interactive Local and Global Features Fusion for Automatic Speech Recognition](https://mlsanity.com/paper/2305.16342/) (2023)
- [Toward a Deeper Understanding: RetNet Viewed through Convolution](https://mlsanity.com/paper/2309.05375/) (2023)
- [PartialFormer: Modeling Part Instead of Whole for Machine Translation](https://mlsanity.com/paper/2310.14921/) (2023)
- [TLM: Token-Level Masking for Transformers](https://mlsanity.com/paper/2310.18738/) (2023)
- [Federated Learning for Cross-block Oil-water Layer Identification](https://mlsanity.com/paper/2112.14359/) (2021)
- [TSNet-SAC: Leveraging Transformers for Efficient Task Scheduling](https://mlsanity.com/paper/2307.07445/) (2023)
- [Unveiling Vulnerability of Self-Attention](https://mlsanity.com/paper/2402.16470/) (2024)
- [Is Smoothness the Key to Robustness? A Comparison of Attention and Convolution Models Using a Novel Metric](https://mlsanity.com/paper/2410.17628/) (2024)
- [MoDA: Modulation Adapter for Fine-Grained Visual Grounding in Instructional MLLMs](https://mlsanity.com/paper/2506.01850/) (2025)

## Abstract (arXiv)

Transformer is an attention-based neural network, which consists of two sublayers, namely, Self-Attention Network (SAN) and Feed-Forward Network (FFN). Existing research explores to enhance the two sublayers separately to improve the capability of Transformer for text representation. In this paper, we present a novel understanding of SAN and FFN as Mask Attention Networks (MANs) and show that they are two special cases of MANs with static mask matrices. However, their static mask matrices limit the capability for localness modeling in text representation learning. We therefore introduce a new layer named dynamic mask attention network (DMAN) with a learnable mask matrix which is able to model localness adaptively. To incorporate advantages of DMAN, SAN, and FFN, we propose a sequential layered structure to combine the three types of layers. Extensive experiments on various tasks, including neural machine translation and text summarization demonstrate that our model outperforms the original Transformer.