mlsanityData to 7 Oct 2026

The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer

arXiv 2204.06457 ↗First posted 13 Apr 2022cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Large multilingual language models such as mBERT or XLM-R enable zero-shot cross-lingual transfer in various IR and NLP tasks. Cao et al. (2020) proposed a data- and compute-efficient method for cross-lingual adjustment of mBERT that uses a small parallel corpus to make embeddings of related words across languages similar to each other. They showed it to be effective in NLI for five European languages. In contrast we experiment with a typologically diverse set of languages (Spanish, Russian, Vietnamese, and Hindi) and extend their original implementations to new tasks (XSR, NER, and QA) and an additional training regime (continual learning). Our study reproduced gains in NLI for four languages, showed improved NER, XSR, and cross-lingual QA results in three languages (though some cross-lingual QA gains were not statistically significant), while mono-lingual QA performance never improved and sometimes degraded. Analysis of distances between contextualized embeddings of related and unrelated words (across languages) showed that fine-tuning leads to "forgetting" some of the cross-lingual alignment information. Based on this observation, we further improved NLI performance using continual learning.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 48 of its 82 references that are in the map. First posted on arXiv 13 Apr 2022.

Builds on

48 of 82 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  7. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  8. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  9. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  10. 2019How multilingual is Multilingual BERT?cited by 642
  11. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  12. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
36 more
  1. 2017Word Translation Without Parallel Datacited by 480
  2. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  3. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  4. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  5. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  6. 2013Exploiting Similarities among Languages for Machine Translationcited by 187
  7. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  8. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 180
  9. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 167
  10. 2019Choosing Transfer Languages for Cross-Lingual Learningcited by 143
  11. 2017A Survey Of Cross-lingual Word Embedding Modelscited by 142
  12. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  13. 2020Multilingual Alignment of Contextual Word Representationscited by 113
  14. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 108
  15. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  16. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  17. 2019Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsingcited by 69
  18. 2019How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representationscited by 63
  19. 2018On the Limitations of Unsupervised Bilingual Dictionary Inductioncited by 61
  20. 2021IsoScore: Measuring the Uniformity of Embedding Space Utilizationcited by 52
  21. 2019How Language-Neutral is Multilingual BERT?cited by 51
  22. 2021AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languagescited by 50
  23. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  24. 2020Inducing Language-Agnostic Multilingual Representationscited by 48
  25. 2021Revisiting the Primacy of English in Zero-shot Cross-lingual Transfercited by 47
  26. 2019Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsingcited by 46
  27. 2020Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answeringcited by 41
  28. 2019Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Frameworkcited by 26
  29. 2020Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Studycited by 24
  30. 2019Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Modelcited by 18
  31. 2019Context-Aware Cross-Lingual Mappingcited by 17
  32. 2021Evaluating Multilingual Text Encoders for Unsupervised Cross-Lingual Retrievalcited by 16
  33. 2020An End-to-End Solution for Named Entity Recognition in eCommerce Searchcited by 6
  34. 2021Can BERT Dig It? -- Named Entity Recognition for Information Retrieval in the Archaeology Domaincited by 5
  35. 2021Investigating Post-pretraining Representation Alignment for Cross-Lingual Question Answeringcited by 3
  36. 2020Unsupervised Evaluation for Question Answering with Transformerscited by 2

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3; 2025: 1.

5 papers.

  1. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  2. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  3. 2023Exploring the Relationship between Alignment and Cross-lingual Transfer in Multilingual Transformerscited by 18
  4. 2025Steering into New Embedding Spaces: Analyzing Cross-Lingual Alignment Induced by Model Interventions in Multilingual Language Modelscited by 10
  5. 2024How Transliterations Improve Crosslingual Alignmentcited by 6