mlsanityData to 14 Oct 2026

On the Cross-lingual Transferability of Monolingual Representations

arXiv 1910.11856 ↗First posted 25 Oct 2019cs.CL cs.AI cs.LGcited by 578

Not on the map. Why?

Abstract (arXiv)

State-of-the-art unsupervised multilingual models (e.g., multilingual BERT) have been shown to generalize in a zero-shot cross-lingual setting. This generalization ability has been attributed to the use of a shared subword vocabulary and joint training across multiple languages giving rise to deep multilingual abstractions. We evaluate this hypothesis by designing an alternative approach that transfers a monolingual model to new languages at the lexical level. More concretely, we first train a transformer-based masked language model on one language, and transfer it to a new language by learning a new embedding matrix with the same masked language modeling objective, freezing parameters of all other layers. This approach does not rely on a shared vocabulary or joint training. However, we show that it is competitive with multilingual BERT on standard cross-lingual classification benchmarks and on a new Cross-lingual Question Answering Dataset (XQuAD). Our results contradict common beliefs of the basis of the generalization ability of multilingual models and suggest that deep monolingual models learn some abstractions that generalize across languages. We also release XQuAD as a more comprehensive cross-lingual benchmark, which comprises 240 paragraphs and 1190 question-answer pairs from SQuAD v1.1 translated into ten languages by professional translators.

Key facts. Cited by 578 papers in the map (data through 14 Oct 2026). First posted on arXiv 25 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 111; 2022: 93; 2023: 101; 2024: 122; 2025: 90; 2026: 61.

578 papers.

Sort:

By area:

  1. 2025Gemma 3 Technical Reportcited by 1,824
  2. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  3. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  4. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  5. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  6. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  7. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 236
  8. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 232
  9. 2021WARP: Word-level Adversarial ReProgrammingcited by 221
  10. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
  11. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  12. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
38 more
  1. 2021MuRIL: Multilingual Representations for Indian Languagescited by 179
  2. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  3. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  4. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 160
  5. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 157
  6. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 151
  7. 2024How do Large Language Models Handle Multilingualism?cited by 150
  8. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  9. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 146
  10. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  11. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  12. 2022mGPT: Few-Shot Learners Go Multilingualcited by 142
  13. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  14. 2021mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Datasetcited by 116
  15. 2023SeaLLMs -- Large Language Models for Southeast Asiacited by 104
  16. 2021Position Information in Transformers: An Overviewcited by 103
  17. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  18. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 92
  19. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  20. 2021MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfercited by 90
  21. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  22. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  23. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  24. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 87
  25. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 84
  26. 2023Spanish Pre-trained BERT Model and Evaluation Datacited by 81
  27. 2021Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translationcited by 74
  28. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  29. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  30. 2022Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answeringcited by 72
  31. 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
  32. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  33. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Modelscited by 67
  34. 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 66
  35. 2021AraT5: Text-to-Text Transformers for Arabic Language Generationcited by 63
  36. 2023Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinationscited by 63
  37. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  38. 2023Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languagescited by 62

50 most cited of 578 All 578 (JSON)