# Alleviating the Inequality of Attention Heads for Neural Machine Translation

arXiv: https://arxiv.org/abs/2009.09672 · HTML: https://mlsanity.com/paper/2009.09672/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Sep 2020.

## Built on by (most cited first, 6 of 6)

- [UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost](https://mlsanity.com/paper/2104.04946/) (2021)
- [Zero-shot Domain Adaptation for Neural Machine Translation with Retrieved Phrase-level Prompts](https://mlsanity.com/paper/2209.11409/) (2022)
- [Not All Attention Is All You Need](https://mlsanity.com/paper/2104.04692/) (2021)
- [Relaxed Attention for Transformer Models](https://mlsanity.com/paper/2209.09735/) (2022)
- [Multi-Objective Decision Transformers for Offline Reinforcement Learning](https://mlsanity.com/paper/2308.16379/) (2023)
- [Only 5% Attention Is All You Need: Efficient Long-range Document-level Neural Machine Translation](https://mlsanity.com/paper/2309.14174/) (2023)

## Abstract (arXiv)

Recent studies show that the attention heads in Transformer are not equal. We relate this phenomenon to the imbalance training of multi-head attention and the model dependence on specific heads. To tackle this problem, we propose a simple masking method: HeadMask, in two specific ways. Experiments show that translation improvements are achieved on multiple language pairs. Subsequent empirical analyses also support our assumption and confirm the effectiveness of the method.