# A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERT

arXiv: https://arxiv.org/abs/2004.14516 · HTML: https://mlsanity.com/paper/2004.14516/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2020.

## Built on by (most cited first, 27 of 27)

- [Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment](https://mlsanity.com/paper/2106.06381/) (2021)
- [Frustratingly Easy Label Projection for Cross-lingual Transfer](https://mlsanity.com/paper/2211.15613/) (2022)
- [arXivEdits: Understanding the Human Revision Process in Scientific Writing](https://mlsanity.com/paper/2210.15067/) (2022)
- [Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings](https://mlsanity.com/paper/2110.02887/) (2021)
- [Neural semi-Markov CRF for Monolingual Word Alignment](https://mlsanity.com/paper/2106.02569/) (2021)
- [Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSA](https://mlsanity.com/paper/2305.14458/) (2023)
- [Word Alignment as Preference for Machine Translation](https://mlsanity.com/paper/2405.09223/) (2024)
- [Enhancing Cross-lingual Sentence Embedding for Low-resource Languages with Word Alignment](https://mlsanity.com/paper/2404.02490/) (2024)
- [WSPAlign: Word Alignment Pre-training via Large-Scale Weakly Supervised Span Prediction](https://mlsanity.com/paper/2306.05644/) (2023)
- [Contextual Label Projection for Cross-Lingual Structured Prediction](https://mlsanity.com/paper/2309.08943/) (2023)
- [Combining Static and Contextualised Multilingual Embeddings](https://mlsanity.com/paper/2203.09326/) (2022)
- [To Translate or Not to Translate: A Systematic Investigation of Translation-Based Cross-Lingual Transfer to Low-Resource Languages](https://mlsanity.com/paper/2311.09404/) (2023)
- [Unbalanced Optimal Transport for Unbalanced Word Alignment](https://mlsanity.com/paper/2306.04116/) (2023)
- [Meeting the Needs of Low-Resource Languages: The Value of Automatic Alignments via Pretrained Models](https://mlsanity.com/paper/2302.07912/) (2023)
- [UDAAN: Machine Learning based Post-Editing tool for Document Translation](https://mlsanity.com/paper/2203.01644/) (2022)
- [Word Alignment in the Era of Deep Learning: A Tutorial](https://mlsanity.com/paper/2212.00138/) (2022)
- [BinaryAlign: Word Alignment as Binary Sequence Labeling](https://mlsanity.com/paper/2407.12881/) (2024)
- [NeoAMT: Neologism-Aware Agentic Machine Translation with Reinforcement Learning](https://mlsanity.com/paper/2601.03790/) (2026)
- [AlignFreeze: Navigating the Impact of Realignment on the Layers of Multilingual Models Across Diverse Languages](https://mlsanity.com/paper/2502.12959/) (2025)
- [Accurate Online Posterior Alignments for Principled Lexically-Constrained Decoding](https://mlsanity.com/paper/2204.00871/) (2022)
- [Entity Tracking via Effective Use of Multi-Task Learning Model and Mention-guided Decoding](https://mlsanity.com/paper/2210.06444/) (2022)
- [Noisy Parallel Data Alignment](https://mlsanity.com/paper/2301.09685/) (2023)
- [OTTAWA: Optimal TransporT Adaptive Word Aligner for Hallucination and Omission Translation Errors Detection](https://mlsanity.com/paper/2406.01919/) (2024)
- [OmniAlign: A Unified Multilingual Aligner for Word and Sentence Alignment](https://mlsanity.com/paper/2608.18474/) (2026)
- [Bilingual Terminology Extraction from Comparable E-Commerce Corpora](https://mlsanity.com/paper/2104.07398/) (2021)
- [Embedding-Enhanced Giza++: Improving Alignment in Low- and High- Resource Scenarios Using Embedding Space Geometry](https://mlsanity.com/paper/2104.08721/) (2021)
- [Extending Word-Level Quality Estimation for Post-Editing Assistance](https://mlsanity.com/paper/2209.11378/) (2022)

## Abstract (arXiv)

We present a novel supervised word alignment method based on cross-language span prediction. We first formalize a word alignment problem as a collection of independent predictions from a token in the source sentence to a span in the target sentence. As this is equivalent to a SQuAD v2.0 style question answering task, we then solve this problem by using multilingual BERT, which is fine-tuned on a manually created gold word alignment data. We greatly improved the word alignment accuracy by adding the context of the token to the question. In the experiments using five word alignment datasets among Chinese, Japanese, German, Romanian, French, and English, we show that the proposed method significantly outperformed previous supervised and unsupervised word alignment methods without using any bitexts for pretraining. For example, we achieved an F1 score of 86.7 for the Chinese-English data, which is 13.3 points higher than the previous state-of-the-art supervised methods.