# Multilingual Sentence Transformer as A Multilingual Word Aligner

arXiv: https://arxiv.org/abs/2301.12140 · HTML: https://mlsanity.com/paper/2301.12140/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 53 references that are in the map.
- First posted on arXiv 28 Jan 2023.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Simple, Scalable Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1909.08478/) (2019)
- [Multilingual Universal Sentence Encoder for Semantic Retrieval](https://mlsanity.com/paper/1907.04307/) (2019)
- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings](https://mlsanity.com/paper/2004.08728/) (2020)
- [On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation](https://mlsanity.com/paper/2106.03164/) (2021)
- [Rapid Adaptation of Neural Machine Translation to New Languages](https://mlsanity.com/paper/1808.04189/) (2018)
- [Jointly Learning to Align and Translate with Transformer Models](https://mlsanity.com/paper/1909.02074/) (2019)
- [Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information](https://mlsanity.com/paper/2010.03142/) (2020)
- [Cross-Lingual Semantic Role Labeling with High-Quality Translated Training Corpus](https://mlsanity.com/paper/2004.06295/) (2020)
- [Weakly Supervised Cross-Lingual Named Entity Recognition via Effective Annotation and Representation Projection](https://mlsanity.com/paper/1707.02483/) (2017)
- [Neural Machine Translation Decoding with Terminology Constraints](https://mlsanity.com/paper/1805.03750/) (2018)
- [Entity Projection via Machine Translation for Cross-Lingual NER](https://mlsanity.com/paper/1909.05356/) (2019)
- [Accurate Word Alignment Induction from Neural Machine Translation](https://mlsanity.com/paper/2004.14837/) (2020)
- [Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution](https://mlsanity.com/paper/2103.06799/) (2021)
- [Adding Interpretable Attention to Neural Translation Models Improves Word Alignment](https://mlsanity.com/paper/1901.11359/) (2019)
- [End-to-End Neural Word Alignment Outperforms GIZA++](https://mlsanity.com/paper/2004.14675/) (2020)
- [Mask-Align: Self-Supervised Neural Word Alignment](https://mlsanity.com/paper/2012.07162/) (2020)
- [Bilingual Lexicon Induction via Unsupervised Bitext Construction and Word Alignment](https://mlsanity.com/paper/2101.00148/) (2021)
- [Contrastive Unsupervised Word Alignment with Non-Local Features](https://mlsanity.com/paper/1410.2082/) (2014)
- [Graph Algorithms for Multiparallel Word Alignment](https://mlsanity.com/paper/2109.06283/) (2021)

## Built on by (most cited first, 8 of 8)

- [To Translate or Not to Translate: A Systematic Investigation of Translation-Based Cross-Lingual Transfer to Low-Resource Languages](https://mlsanity.com/paper/2311.09404/) (2023)
- [BinaryAlign: Word Alignment as Binary Sequence Labeling](https://mlsanity.com/paper/2407.12881/) (2024)
- [The Devil Is in the Word Alignment Details: On Translation-Based Cross-Lingual Transfer for Token Classification Tasks](https://mlsanity.com/paper/2505.10507/) (2025)
- [Cross-lingual Contextualized Phrase Retrieval](https://mlsanity.com/paper/2403.16820/) (2024)
- [OmniAlign: A Unified Multilingual Aligner for Word and Sentence Alignment](https://mlsanity.com/paper/2608.18474/) (2026)
- [Scaling Unsupervised Word Alignment to Documents via Structural Constraints](https://mlsanity.com/paper/2608.21023/) (2026)
- [SMCLM: Semantically Meaningful Causal Language Modeling for Autoregressive Paraphrase Generation](https://mlsanity.com/paper/2507.03415/) (2025)
- [TransAlign: Machine Translation Encoders are Strong Word Aligners, Too](https://mlsanity.com/paper/2510.27337/) (2025)

## Abstract (arXiv)

Multilingual pretrained language models (mPLMs) have shown their effectiveness in multilingual word alignment induction. However, these methods usually start from mBERT or XLM-R. In this paper, we investigate whether multilingual sentence Transformer LaBSE is a strong multilingual word aligner. This idea is non-trivial as LaBSE is trained to learn language-agnostic sentence-level embeddings, while the alignment extraction task requires the more fine-grained word-level embeddings to be language-agnostic. We demonstrate that the vanilla LaBSE outperforms other mPLMs currently used in the alignment task, and then propose to finetune LaBSE on parallel corpus for further improvement. Experiment results on seven language pairs show that our best aligner outperforms previous state-of-the-art models of all varieties. In addition, our aligner supports different language pairs in a single model, and even achieves new state-of-the-art on zero-shot language pairs that does not appear in the finetuning process.