mlsanityData to 14 Oct 2026

WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models

arXiv 2112.06598 ↗First posted 13 Dec 2021cs.CLcited by 89

Not on the map. Why?

Abstract (arXiv)

Large pretrained language models (LMs) have become the central building block of many NLP applications. Training these models requires ever more computational resources and most of the existing models are trained on English text only. It is exceedingly expensive to train these models in other languages. To alleviate this problem, we introduce a novel method -- called WECHSEL -- to efficiently and effectively transfer pretrained LMs to new languages. WECHSEL can be applied to any model which uses subword-based tokenization and learns an embedding for each subword. The tokenizer of the source model (in English) is replaced with a tokenizer in the target language and token embeddings are initialized such that they are semantically similar to the English tokens by utilizing multilingual static word embeddings covering English and the target language. We use WECHSEL to transfer the English RoBERTa and GPT-2 models to four languages (French, German, Chinese and Swahili). We also study the benefits of our method on very low-resource languages. WECHSEL improves over proposed methods for cross-lingual parameter transfer and outperforms models of comparable size trained from scratch with up to 64x less training effort. Our method makes training large language models for new languages more accessible and less damaging to the environment. We make our code and models publicly available.

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Builds on 36 of its 63 references that are in the map. First posted on arXiv 13 Dec 2021.

Builds on

36 of 63 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  9. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  10. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  12. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
24 more
  1. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  2. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  3. 2019How multilingual is Multilingual BERT?cited by 645
  4. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  5. 2017Word Translation Without Parallel Datacited by 487
  6. 2019CamemBERT: a Tasty French Language Modelcited by 368
  7. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 362
  8. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  9. 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
  10. 2016Transfer Learning for Low-Resource Neural Machine Translationcited by 188
  11. 2018Adversarial Removal of Demographic Attributes from Text Datacited by 170
  12. 2018A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddingscited by 146
  13. 2020German's Next Language Modelcited by 144
  14. 2020Pre-training via Paraphrasingcited by 92
  15. 2018Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterioncited by 70
  16. 2021Few-Shot Question Answering by Pretraining Span Selectioncited by 62
  17. 2017Transfer Learning across Low-Resource, Related Languages for Neural Machine Translationcited by 58
  18. 2020As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languagescited by 54
  19. 2020What the [MASK]? Making Sense of Language-Specific BERT Modelscited by 44
  20. 2020From English To Foreign Languages: Transferring Pre-trained Language Modelscited by 37
  21. 2021Societal Biases in Retrieved Contents: Measurement Framework and Adversarial Mitigation for BERT Rankerscited by 35
  22. 2020Do Neural Ranking Models Intensify Gender Bias?cited by 29
  23. 2016Learning Crosslingual Word Embeddings without Bilingual Corporacited by 12
  24. 2019Regularization Advantages of Multilingual Neural Language Models for Low Resource Domainscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 14; 2024: 26; 2025: 23; 2026: 21.

89 papers.

Sort:

By area:

  1. 2025SuperBPE: Space Travel for Language Modelscited by 63
  2. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 41
  3. 2022History Compression via Language Models in Reinforcement Learningcited by 39
  4. 2023OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretrainingcited by 37
  5. 2024Zero-Shot Tokenizer Transfercited by 31
  6. 2024RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanizationcited by 29
  7. 2023Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learningcited by 26
  8. 2022GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Costcited by 24
  9. 2024How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?cited by 22
  10. 2024Exploring Design Choices for Building Language-Specific LLMscited by 22
  11. 2024An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Modelscited by 20
  12. 2024An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inferencecited by 18
38 more
  1. 2023Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languagescited by 16
  2. 2024Retrofitting Large Language Models with Dynamic Tokenizationcited by 16
  3. 2023Transfer to a Low-Resource Language via Close Relatives: The Case Study on Faroesecited by 15
  4. 2023Language Models for German Text Simplification: Overcoming Parallel Data Scarcity through Style-specific Pre-trainingcited by 15
  5. 2024The Zeno's Paradox of `Low-Resource' Languagescited by 12
  6. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 12
  7. 2023Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Modelscited by 11
  8. 2024Large Vocabulary Size Improves Large Language Modelscited by 11
  9. 2025TokAlign: Efficient Vocabulary Adaptation via Token Alignmentcited by 11
  10. 2025Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindicited by 10
  11. 2022Cross-lingual Lifelong Learningcited by 9
  12. 2024Targeted Multilingual Adaptation for Low-resource Language Familiescited by 9
  13. 2024Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?cited by 9
  14. 2025Domain Adaptation of Foundation LLMs for e-Commercecited by 9
  15. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 9
  16. 2023Team QUST at SemEval-2023 Task 3: A Comprehensive Study of Monolingual and Multilingual Approaches for Detecting Online News Genre, Framing and Persuasion Techniquescited by 8
  17. 2024Efficient Continual Pre-training of LLMs for Low-resource Languagescited by 8
  18. 2025Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansioncited by 8
  19. 2025FLEXITOKENS: Flexible Tokenization for Evolving Language Modelscited by 8
  20. 2022Automatic Readability Assessment of German Sentences with Transformer Ensemblescited by 7
  21. 2025Sampling from Your Language Model One Byte at a Timecited by 7
  22. 2023Distilling Efficient Language-Specific Models for Cross-Lingual Transfercited by 6
  23. 2024Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Modelscited by 6
  24. 2024Adapting Chat Language Models Using Only Target Unlabeled Language Datacited by 6
  25. 2025Token Distillation: Attention-aware Input Embeddings For New Tokenscited by 6
  26. 2025Model-Aware Tokenizer Transfercited by 6
  27. 2023Architectural Sweet Spots for Modeling Human Label Variation by the Example of Argument Quality: It's Best to Relate Perspectives!cited by 5
  28. 2024Bilingual Adaptation of Monolingual Foundation Modelscited by 5
  29. 2024Generative Model for Less-Resourced Language with 1 billion parameterscited by 5
  30. 2025Bielik v3 Small: Technical Reportcited by 5
  31. 2024German Text Simplification: Finetuning Large Language Models with Semi-Synthetic Datacited by 4
  32. 2024UniBridge: A Unified Approach to Cross-Lingual Transfer Learning for Low-Resource Languagescited by 4
  33. 2024Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guidecited by 4
  34. 2024Transducer-Llama: Integrating LLMs into Streamable Transducer-based Speech Recognitioncited by 4
  35. 2025Sherkala-Chat: Building a State-of-the-Art LLM for Kazakh in a Moderately Resourced Settingcited by 4
  36. 2025Achieving Tokenizer Flexibility in Language Models through Heuristic Adaptation and Supertoken Learningcited by 4
  37. 2025Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuitcited by 4
  38. 2025HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initializationcited by 3

50 most cited of 89 All 89 (JSON)