# Mask-Align: Self-Supervised Neural Word Alignment

arXiv: https://arxiv.org/abs/2012.07162 · HTML: https://mlsanity.com/paper/2012.07162/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Dec 2020.

## Built on by (most cited first, 20 of 20)

- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [TranSmart: A Practical Interactive Machine Translation System](https://mlsanity.com/paper/2105.13072/) (2021)
- [Inducing and Using Alignments for Transition-based AMR Parsing](https://mlsanity.com/paper/2205.01464/) (2022)
- [Integrating Vectorized Lexical Constraints for Neural Machine Translation](https://mlsanity.com/paper/2203.12210/) (2022)
- [Multilingual Sentence Transformer as A Multilingual Word Aligner](https://mlsanity.com/paper/2301.12140/) (2023)
- [Anticipation-Free Training for Simultaneous Machine Translation](https://mlsanity.com/paper/2201.12868/) (2022)
- [The Devil is in the Details: On the Pitfalls of Vocabulary Selection in Neural Machine Translation](https://mlsanity.com/paper/2205.06618/) (2022)
- [Weakly Supervised Face Naming with Symmetry-Enhanced Contrastive Loss](https://mlsanity.com/paper/2210.08957/) (2022)
- [Word Alignment in the Era of Deep Learning: A Tutorial](https://mlsanity.com/paper/2212.00138/) (2022)
- [BinaryAlign: Word Alignment as Binary Sequence Labeling](https://mlsanity.com/paper/2407.12881/) (2024)
- [Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment](https://mlsanity.com/paper/2210.04141/) (2022)
- [Prompting Neural Machine Translation with Translation Memories](https://mlsanity.com/paper/2301.05380/) (2023)
- [Noisy Parallel Data Alignment](https://mlsanity.com/paper/2301.09685/) (2023)
- [Pluggable Neural Machine Translation Models via Memory-augmented Adapters](https://mlsanity.com/paper/2307.06029/) (2023)
- [Towards General Error Diagnosis via Behavioral Testing in Machine Translation](https://mlsanity.com/paper/2310.13362/) (2023)
- [Towards Privacy-Preserving Machine Translation at the Inference Stage: A New Task and Benchmark](https://mlsanity.com/paper/2603.14756/) (2026)
- [Leveraging Neural Machine Translation for Word Alignment](https://mlsanity.com/paper/2103.17250/) (2021)
- [Cross-lingual AMR Aligner: Paying Attention to Cross-Attention](https://mlsanity.com/paper/2206.07587/) (2022)
- [Third-Party Aligner for Neural Word Alignments](https://mlsanity.com/paper/2211.04198/) (2022)
- [From Attention Masks to Inert Zero-Vector Tokens: OAttention and O-Closure for Token Dynamics](https://mlsanity.com/paper/2608.21174/) (2026)

## Abstract (arXiv)

Word alignment, which aims to align translationally equivalent words between source and target sentences, plays an important role in many natural language processing tasks. Current unsupervised neural alignment methods focus on inducing alignments from neural machine translation models, which does not leverage the full context in the target sequence. In this paper, we propose Mask-Align, a self-supervised word alignment model that takes advantage of the full context on the target side. Our model masks out each target token and predicts it conditioned on both source and the remaining target tokens. This two-step process is based on the assumption that the source token contributing most to recovering the masked target token should be aligned. We also introduce an attention variant called leaky attention, which alleviates the problem of unexpected high cross-attention weights on special tokens such as periods. Experiments on four language pairs show that our model outperforms previous unsupervised neural aligners and obtains new state-of-the-art results.