# Multilingual BERT Post-Pretraining Alignment

arXiv: https://arxiv.org/abs/2010.12547 · HTML: https://mlsanity.com/paper/2010.12547/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2020.

## Built on by (most cited first, 30 of 30)

- [XLM-E: Cross-lingual Language Model Pre-training via ELECTRA](https://mlsanity.com/paper/2106.16138/) (2021)
- [Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructions](https://mlsanity.com/paper/2305.15083/) (2023)
- [Improved Text Classification via Contrastive Adversarial Training](https://mlsanity.com/paper/2107.10137/) (2021)
- [OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining](https://mlsanity.com/paper/2311.08849/) (2023)
- [Understanding Cross-Lingual Alignment -- A Survey](https://mlsanity.com/paper/2404.06228/) (2024)
- [Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment](https://mlsanity.com/paper/2311.08089/) (2023)
- [TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning](https://mlsanity.com/paper/2111.04198/) (2021)
- [Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training](https://mlsanity.com/paper/2104.08645/) (2021)
- [PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignment](https://mlsanity.com/paper/2407.16222/) (2024)
- [XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge](https://mlsanity.com/paper/2109.12573/) (2021)
- [Exploring the Relationship between Alignment and Cross-lingual Transfer in Multilingual Transformers](https://mlsanity.com/paper/2306.02790/) (2023)
- [Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learning](https://mlsanity.com/paper/2110.02600/) (2021)
- [Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment](https://mlsanity.com/paper/2406.19759/) (2024)
- [Concept Space Alignment in Multilingual LLMs](https://mlsanity.com/paper/2410.01079/) (2024)
- [XeroAlign: Zero-Shot Cross-lingual Transformer Alignment](https://mlsanity.com/paper/2105.02472/) (2021)
- [CINO: A Chinese Minority Pre-trained Language Model](https://mlsanity.com/paper/2202.13558/) (2022)
- [Towards Better Chinese-centric Neural Machine Translation for Low-resource Languages](https://mlsanity.com/paper/2204.04344/) (2022)
- [CrossAligner & Co: Zero-Shot Transfer Methods for Task-Oriented Cross-lingual Natural Language Understanding](https://mlsanity.com/paper/2203.09982/) (2022)
- [How Transliterations Improve Crosslingual Alignment](https://mlsanity.com/paper/2409.17326/) (2024)
- [A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning](https://mlsanity.com/paper/2210.09934/) (2022)
- [Cross-Lingual Text Classification with Multilingual Distillation and Zero-Shot-Aware Training](https://mlsanity.com/paper/2202.13654/) (2022)
- [Multi-level Distillation of Semantic Knowledge for Pre-training Multilingual Language Model](https://mlsanity.com/paper/2211.01200/) (2022)
- [TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2401.06620/) (2024)
- [Improving Zero-Shot Cross-Lingual Transfer via Progressive Code-Switching](https://mlsanity.com/paper/2406.13361/) (2024)
- [Momentum Contrastive Pre-training for Question Answering](https://mlsanity.com/paper/2212.05762/) (2022)
- [LangSAMP: Language-Script Aware Multilingual Pretraining](https://mlsanity.com/paper/2409.18199/) (2024)
- [AlignFreeze: Navigating the Impact of Realignment on the Layers of Multilingual Models Across Diverse Languages](https://mlsanity.com/paper/2502.12959/) (2025)
- [Debiasing Multilingual LLMs in Cross-lingual Latent Space](https://mlsanity.com/paper/2508.17948/) (2025)
- [When do Contrastive Word Alignments Improve Many-to-many Neural Machine Translation?](https://mlsanity.com/paper/2204.12165/) (2022)
- [Natural Language based Context Modeling and Reasoning for Ubiquitous Computing with Large Language Models: A Tutorial](https://mlsanity.com/paper/2309.15074/) (2023)

## Abstract (arXiv)

We propose a simple method to align multilingual contextual embeddings as a post-pretraining step for improved zero-shot cross-lingual transferability of the pretrained models. Using parallel data, our method aligns embeddings on the word level through the recently proposed Translation Language Modeling objective as well as on the sentence level via contrastive learning and random input shuffling. We also perform sentence-level code-switching with English when finetuning on downstream tasks. On XNLI, our best model (initialized from mBERT) improves over mBERT by 4.7% in the zero-shot setting and achieves comparable result to XLM for translate-train while using less than 18% of the same parallel data and 31% less model parameters. On MLQA, our model outperforms XLM-R_Base that has 57% more parameters than ours.