# Multi-Granularity Self-Attention for Neural Machine Translation

arXiv: https://arxiv.org/abs/1909.02222 · HTML: https://mlsanity.com/paper/1909.02222/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Sep 2019.

## Built on by (most cited first, 10 of 10)

- [Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Research](https://mlsanity.com/paper/2110.08975/) (2021)
- [Learning Multiscale Transformer Models for Sequence Generation](https://mlsanity.com/paper/2206.09337/) (2022)
- [TranSFormer: Slow-Fast Transformer for Machine Translation](https://mlsanity.com/paper/2305.16982/) (2023)
- [Enhancing Neural Machine Translation with Semantic Units](https://mlsanity.com/paper/2310.11360/) (2023)
- [Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models](https://mlsanity.com/paper/2310.16240/) (2023)
- [Enhancing the Transformer Decoder with Transition-based Syntax](https://mlsanity.com/paper/2101.12640/) (2021)
- [Continuous Decomposition of Granularity for Neural Paraphrase Generation](https://mlsanity.com/paper/2209.01765/) (2022)
- [EIT: Enhanced Interactive Transformer](https://mlsanity.com/paper/2212.10197/) (2022)
- [Towards Reliable Neural Machine Translation with Consistency-Aware Meta-Learning](https://mlsanity.com/paper/2303.10966/) (2023)
- [Deterministic Reversible Data Augmentation for Neural Machine Translation](https://mlsanity.com/paper/2406.02517/) (2024)

## Abstract (arXiv)

Current state-of-the-art neural machine translation (NMT) uses a deep multi-head self-attention network with no explicit phrase information. However, prior work on statistical machine translation has shown that extending the basic translation unit from words to phrases has produced substantial improvements, suggesting the possibility of improving NMT performance from explicit modeling of phrases. In this work, we present multi-granularity self-attention (Mg-Sa): a neural network that combines multi-head self-attention and phrase modeling. Specifically, we train several attention heads to attend to phrases in either n-gram or syntactic formalism. Moreover, we exploit interactions among phrases to enhance the strength of structure modeling - a commonly-cited weakness of self-attention. Experimental results on WMT14 English-to-German and NIST Chinese-to-English translation tasks show the proposed approach consistently improves performance. Targeted linguistic analysis reveals that Mg-Sa indeed captures useful phrase information at various levels of granularities.