mlsanityData to 7 Oct 2026

Multilingual Alignment of Contextual Word Representations

arXiv 2002.03518 ↗First posted 10 Feb 2020cs.CL cs.LGcited by 113

Not on the map. Why?

Abstract (arXiv)

We propose procedures for evaluating and strengthening contextual embedding alignment and show that they are useful in analyzing and improving multilingual BERT. In particular, after our proposed alignment procedure, BERT exhibits significantly improved zero-shot performance on XNLI compared to the base model, remarkably matching pseudo-fully-supervised translate-train models for Bulgarian and Greek. Further, to measure the degree of alignment, we introduce a contextual version of word retrieval and show that it correlates well with downstream zero-shot transfer. Using this word retrieval task, we also analyze BERT and find that it exhibits systematic deficiencies, e.g. worse alignment for open-class parts-of-speech and word pairs written in different scripts, that are corrected by the alignment procedure. These results support contextual alignment as a useful concept for understanding large multilingual pre-trained models.

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 33; 2022: 23; 2023: 13; 2024: 28; 2025: 8; 2026: 8.

113 papers.

Sort:

By area:

  1. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  2. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  3. 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 124
  4. 2022The Geometry of Multilingual Language Model Representationscited by 94
  5. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 94
  6. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  7. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  8. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  9. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  10. 2021Revisiting the Primacy of English in Zero-shot Cross-lingual Transfercited by 47
  11. 2021MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairscited by 41
  12. 2021Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignmentcited by 41
38 more
  1. 2021When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfercited by 41
  2. 2024Probing the Emergence of Cross-lingual Alignment during LLM Trainingcited by 38
  3. 2022Speaking Multiple Languages Affects the Moral Bias of Language Modelscited by 34
  4. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  5. 2023Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignmentcited by 33
  6. 2022English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddingscited by 26
  7. 2021Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factorscited by 25
  8. 2022Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languagescited by 25
  9. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  10. 2021On Cross-Lingual Retrieval with Multilingual Text Encoderscited by 22
  11. 2023Why Does Zero-Shot Cross-Lingual Generation Fail? An Explanation and a Solutioncited by 22
  12. 2024Unveiling Linguistic Regions in Large Language Modelscited by 22
  13. 2021Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Trainingcited by 21
  14. 2021Specializing Multilingual Language Models: An Empirical Studycited by 21
  15. 2023InstructAlign: High-and-Low Resource Language Alignment via Continual Crosslingual Instruction Tuningcited by 21
  16. 2024PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignmentcited by 20
  17. 2021Exploiting Language Relatedness for Low Web-Resource Language Model Adaptation: An Indic Languages Studycited by 19
  18. 2021Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddingscited by 19
  19. 2021mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Modelscited by 19
  20. 2023Exploring the Relationship between Alignment and Cross-lingual Transfer in Multilingual Transformerscited by 18
  21. 2022Parameter-Efficient Neural Reranking for Cross-Lingual and Multilingual Retrievalcited by 17
  22. 2021Evaluating Multilingual Text Encoders for Unsupervised Cross-Lingual Retrievalcited by 16
  23. 2021Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systemscited by 16
  24. 2021Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learningcited by 16
  25. 2022USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translationcited by 15
  26. 2022Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Modelscited by 14
  27. 2024Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignmentcited by 14
  28. 2024Exploring Alignment in Shared Cross-lingual Spacescited by 13
  29. 2021Combining Static Word Embeddings and Contextual Representations for Bilingual Lexicon Inductioncited by 12
  30. 2022CINO: A Chinese Minority Pre-trained Language Modelcited by 11
  31. 2024Cross-Lingual Transfer Learning for Speech Translationcited by 11
  32. 2025False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Modelscited by 11
  33. 2022Analyzing Gender Representation in Multilingual Modelscited by 10
  34. 2022Cross-Linguistic Syntactic Difference in Multilingual BERT: How Good is It and How Does It Affect Transfer?cited by 10
  35. 2021Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Miningcited by 9
  36. 2021El Volumen Louder Por Favor: Code-switching in Task-oriented Semantic Parsingcited by 8
  37. 2021Bootstrapping Multilingual AMR with Contextual Word Alignmentscited by 8
  38. 2022Combining Static and Contextualised Multilingual Embeddingscited by 8

50 most cited of 113 All 113 (JSON)