# Do Explicit Alignments Robustly Improve Multilingual Encoders?

arXiv: https://arxiv.org/abs/2010.02537 · HTML: https://mlsanity.com/paper/2010.02537/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 7 of 7)

- [Reinforced Iterative Knowledge Distillation for Cross-Lingual Named Entity Recognition](https://mlsanity.com/paper/2106.00241/) (2021)
- [A Dual-Contrastive Framework for Low-Resource Cross-Lingual Named Entity Recognition](https://mlsanity.com/paper/2204.00796/) (2022)
- [Constrained Density Matching and Modeling for Cross-lingual Alignment of Contextualized Representations](https://mlsanity.com/paper/2201.13429/) (2022)
- [Mismatching-Aware Unsupervised Translation Quality Estimation For Low-Resource Languages](https://mlsanity.com/paper/2208.00463/) (2022)
- [DiTTO: A Feature Representation Imitation Approach for Improving Cross-Lingual Transfer](https://mlsanity.com/paper/2303.02357/) (2023)
- [Bridging Linguistic Gaps: Cross-Lingual Mapping in Pre-Training and Dataset for Enhanced Multilingual LLM Performance](https://mlsanity.com/paper/2604.10590/) (2026)
- [Rethinking what Matters: Effective and Robust Multilingual Realignment for Low-Resource Languages](https://mlsanity.com/paper/2511.06497/) (2025)

## Abstract (arXiv)

Multilingual BERT (mBERT), XLM-RoBERTa (XLMR) and other unsupervised multilingual encoders can effectively learn cross-lingual representation. Explicit alignment objectives based on bitexts like Europarl or MultiUN have been shown to further improve these representations. However, word-level alignments are often suboptimal and such bitexts are unavailable for many languages. In this paper, we propose a new contrastive alignment objective that can better utilize such signal, and examine whether these previous alignment methods can be adapted to noisier sources of aligned data: a randomly sampled 1 million pair subset of the OPUS collection. Additionally, rather than report results on a single dataset with a single model run, we report the mean and standard derivation of multiple runs with different seeds, on four datasets and tasks. Our more extensive analysis finds that, while our new objective outperforms previous work, overall these methods do not improve performance with a more robust evaluation framework. Furthermore, the gains from using a better underlying model eclipse any benefits from alignment training. These negative results dictate more care in evaluating these methods and suggest limitations in applying explicit alignment objectives.