mlsanityData to 7 Oct 2026

Multilingual BERT Post-Pretraining Alignment

arXiv 2010.12547 ↗First posted 23 Oct 2020cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

We propose a simple method to align multilingual contextual embeddings as a post-pretraining step for improved zero-shot cross-lingual transferability of the pretrained models. Using parallel data, our method aligns embeddings on the word level through the recently proposed Translation Language Modeling objective as well as on the sentence level via contrastive learning and random input shuffling. We also perform sentence-level code-switching with English when finetuning on downstream tasks. On XNLI, our best model (initialized from mBERT) improves over mBERT by 4.7% in the zero-shot setting and achieves comparable result to XLM for translate-train while using less than 18% of the same parallel data and 31% less model parameters. On MLQA, our model outperforms XLM-R_Base that has 57% more parameters than ours.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 7; 2022: 8; 2023: 5; 2024: 8; 2025: 2.

30 papers.

Sort:

By area:

  1. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  2. 2023Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructionscited by 55
  3. 2021Improved Text Classification via Contrastive Adversarial Trainingcited by 38
  4. 2023OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretrainingcited by 37
  5. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  6. 2023Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignmentcited by 33
  7. 2021TaCL: Improving BERT Pre-training with Token-aware Contrastive Learningcited by 32
  8. 2021Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Trainingcited by 21
  9. 2024PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignmentcited by 20
  10. 2021XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledgecited by 18
  11. 2023Exploring the Relationship between Alignment and Cross-lingual Transfer in Multilingual Transformerscited by 18
  12. 2021Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learningcited by 16
18 more
  1. 2024Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignmentcited by 14
  2. 2024Concept Space Alignment in Multilingual LLMscited by 13
  3. 2021XeroAlign: Zero-Shot Cross-lingual Transformer Alignmentcited by 11
  4. 2022CINO: A Chinese Minority Pre-trained Language Modelcited by 11
  5. 2022Towards Better Chinese-centric Neural Machine Translation for Low-resource Languagescited by 11
  6. 2022CrossAligner & Co: Zero-Shot Transfer Methods for Task-Oriented Cross-lingual Natural Language Understandingcited by 7
  7. 2024How Transliterations Improve Crosslingual Alignmentcited by 6
  8. 2022A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learningcited by 4
  9. 2022Cross-Lingual Text Classification with Multilingual Distillation and Zero-Shot-Aware Trainingcited by 3
  10. 2022Multi-level Distillation of Semantic Knowledge for Pre-training Multilingual Language Modelcited by 3
  11. 2024TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Modelscited by 3
  12. 2024Improving Zero-Shot Cross-Lingual Transfer via Progressive Code-Switchingcited by 3
  13. 2022Momentum Contrastive Pre-training for Question Answeringcited by 2
  14. 2024LangSAMP: Language-Script Aware Multilingual Pretrainingcited by 2
  15. 2025AlignFreeze: Navigating the Impact of Realignment on the Layers of Multilingual Models Across Diverse Languagescited by 2
  16. 2025Debiasing Multilingual LLMs in Cross-lingual Latent Spacecited by 2
  17. 2022When do Contrastive Word Alignments Improve Many-to-many Neural Machine Translation?cited by 1
  18. 2023Natural Language based Context Modeling and Reasoning for Ubiquitous Computing with Large Language Models: A Tutorialcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Benchmarking Cross-Lingual Semantic Alignment in Multilingual Embeddings
  2. 2021First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT
  3. 2021Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
  4. 2021A Primer on Pretrained Multilingual Language Models
  5. 2023Multilingual Sentence Transformer as A Multilingual Word Aligner
  6. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corpora
  7. 2023Dual-Alignment Pre-training for Cross-lingual Sentence Embedding
  8. 2026Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?