mlsanityData to 7 Oct 2026

From English To Foreign Languages: Transferring Pre-trained Language Models

arXiv 2002.07306 ↗First posted 18 Feb 2020cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

Pre-trained models have demonstrated their effectiveness in many downstream natural language processing (NLP) tasks. The availability of multilingual pre-trained models enables zero-shot transfer of NLP tasks from high resource languages to low resource ones. However, recent research in improving pre-trained models focuses heavily on English. While it is possible to train the latest neural architectures for other languages from scratch, it is undesirable due to the required amount of compute. In this work, we tackle the problem of transferring an existing pre-trained model from English to other languages under a limited computational budget. With a single GPU, our approach can obtain a foreign BERT base model within a day and a foreign BERT large within two days. Furthermore, evaluating our models on six languages, we demonstrate that our models are better than multilingual BERT on two zero-shot tasks: natural language inference and dependency parsing.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 5; 2023: 4; 2024: 8; 2025: 11; 2026: 3.

37 papers.

Sort:

  1. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  2. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 90
  3. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  4. 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
  5. 2025SuperBPE: Space Travel for Language Modelscited by 58
  6. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 40
  7. 2023OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretrainingcited by 37
  8. 2024Zero-Shot Tokenizer Transfercited by 30
  9. 2022GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Costcited by 24
  10. 2022Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Trainingcited by 24
  11. 2024An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Modelscited by 20
  12. 2024An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inferencecited by 18
25 more
  1. 2021Subword Mapping and Anchoring across Languagescited by 12
  2. 2025TokAlign: Efficient Vocabulary Adaptation via Token Alignmentcited by 11
  3. 2022Oolong: Investigating What Makes Transfer Learning Hard with Controlled Studiescited by 9
  4. 2024Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?cited by 9
  5. 2025Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansioncited by 8
  6. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 8
  7. 2025Sampling from Your Language Model One Byte at a Timecited by 7
  8. 2023Distilling Efficient Language-Specific Models for Cross-Lingual Transfercited by 6
  9. 2025Token Distillation: Attention-aware Input Embeddings For New Tokenscited by 6
  10. 2024Exploring Tokenization Strategies and Vocabulary Sizes for Enhanced Arabic Language Modelscited by 5
  11. 2024ReTok: Replacing Tokenizer to Enhance Representation Efficiency in Large Language Modelcited by 5
  12. 2021On the Universality of Deep Contextual Language Modelscited by 4
  13. 2025Bielik v3 Small: Technical Reportcited by 4
  14. 2021Cross-lingual Transferring of Pre-trained Contextualized Language Modelscited by 3
  15. 2025HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initializationcited by 3
  16. 2025Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfercited by 2
  17. 2025Cross-Lingual Optimization for Language Transfer in Large Language Modelscited by 2
  18. 2021Identifying the Limits of Cross-Domain Knowledge Transfer for Pretrained Modelscited by 1
  19. 2026Advancing Polish Language Modeling through Tokenizer Optimization in the Bielik v3 7B and 11B Seriescited by 1
  20. 2026TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignmentcited by 1
  21. 2021Graph Convolutional Network for Swahili News Classificationcited by 0
  22. 2023FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Modelscited by 0
  23. 2024Transferring BERT Capabilities from High-Resource to Low-Resource Languages Using Vocabulary Matchingcited by 0
  24. 2024Bailong: Bilingual Transfer Learning based on QLoRA and Zip-tie Embeddingcited by 0
  25. 2026Effective vocabulary expansion of multilingual language models for extremely low-resource languagescited by 0