mlsanityData to 7 Oct 2026

Cross-lingual Transfer of Monolingual Models

arXiv 2109.07348 ↗First posted 15 Sep 2021cs.CL cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Recent studies in zero-shot cross-lingual learning using multilingual models have falsified the previous hypothesis that shared vocabulary and joint pre-training are the keys to cross-lingual generalization. Inspired by this advancement, we introduce a cross-lingual transfer method for monolingual models based on domain adaptation. We study the effects of such transfer from four different languages to English. Our experimental results on GLUE show that the transferred models outperform the native English model independently of the source language. After probing the English linguistic knowledge encoded in the representations before and after transfer, we find that semantic information is retained from the source language, while syntactic information is learned during transfer. Additionally, the results of evaluating the transferred models in source language tasks reveal that their performance in the source domain deteriorates after transfer.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 36 of its 47 references that are in the map. First posted on arXiv 15 Sep 2021.

Builds on

36 of 47 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  8. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  9. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  10. 2019How multilingual is Multilingual BERT?cited by 642
  11. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  12. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
24 more
  1. 2018Gender Bias in Coreference Resolutioncited by 438
  2. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  3. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  4. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  5. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 359
  6. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  7. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 269
  8. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  9. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 180
  10. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  11. 2017A Survey Of Cross-lingual Word Embedding Modelscited by 142
  12. 2020ProGen: Language Modeling for Protein Generationcited by 127
  13. 2019Multilingual is not enough: BERT for Finnishcited by 118
  14. 2019BERTje: A Dutch BERT Modelcited by 109
  15. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 108
  16. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  17. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
  18. 2020BERTology Meets Biology: Interpreting Attention in Protein Language Modelscited by 85
  19. 2019Learning and Evaluating General Linguistic Intelligencecited by 78
  20. 2020As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languagescited by 54
  21. 2017Learning Joint Multilingual Sentence Representations with Neural Machine Translationcited by 52
  22. 2020Playing with Words at the National Library of Sweden -- Making a Swedish BERTcited by 35
  23. 2021Adapting Monolingual Models: Data can be Scarce when Language Similarity is Highcited by 12
  24. 2020Why Not Simply Translate? A First Swedish Evaluation Benchmark for Semantic Similaritycited by 3

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 4; 2024: 5; 2025: 3.

14 papers.

Sort:

By area:

  1. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 90
  2. 2024Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLPcited by 35
  3. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27
  4. 2023Transfer to a Low-Resource Language via Close Relatives: The Case Study on Faroesecited by 15
  5. 2023Tik-to-Tok: Translating Language Models One Token at a Time: An Embedding Initialization Strategy for Efficient Language Adaptationcited by 11
  6. 2022Oolong: Investigating What Makes Transfer Learning Hard with Controlled Studiescited by 9
  7. 2024Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?cited by 9
  8. 2023DUMB: A Benchmark for Smart Evaluation of Dutch Modelscited by 6
  9. 2024Cross-lingual transfer of multilingual models on low resource African Languagescited by 5
  10. 2025Model-Aware Tokenizer Transfercited by 5
  11. 2025Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfercited by 2
  12. 2024Cross-lingual Transfer or Machine Translation? On Data Augmentation for Monolingual Semantic Textual Similaritycited by 1
2 more
  1. 2023Knowledge from Large-Scale Protein Contact Prediction Models Can Be Transferred to the Data-Scarce RNA Contact Prediction Taskcited by 0
  2. 2025Scaling Laws for Conditional Emergence of Multilingual Image Captioning via Generalization from Translationcited by 0