mlsanityData to 14 Oct 2026

Multilingual Knowledge Transfer under Data Constraints via Lexical Interventions

arXiv 2605.23885 ↗First posted 22 May 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Cross-lingual knowledge transfer is critical for building high-performing multilingual language models for languages with insufficient training data. When target language data is scarce, the knowledge required for many downstream tasks involving scientific reasoning, commonsense inference, and world knowledge must be acquired primarily from the high-resource language, making effective knowledge transfer essential. Existing methods for improving such cross-lingual knowledge transfer require large amounts of parallel data, translation systems, auxiliary models, or additional training stages that are largely unavailable for many languages. We propose LINK - a data-level intervention method that improves knowledge transfer during model pretraining through lexical substitutions in high-resource part of pretraining data using bilingual vocabularies. For a given replacement ratio, randomly selected words in a portion of the high-resource (English) training corpus are swapped with their word-level translations, requiring no additional model training and only a bilingual vocabulary, which can be obtained at near-zero cost for virtually any language. Evaluation on eight languages across five model sizes shows notable improvements on downstream tasks in the target language, with up to a 2x speedup in training to reach equivalent performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 39 references that are in the map. First posted on arXiv 22 May 2026.

Builds on

32 of 39 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  8. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  10. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  11. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  12. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
20 more
  1. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  2. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  3. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  4. 2018Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relationscited by 201
  5. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 181
  6. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 143
  7. 2017Data Augmentation for Low-Resource Neural Machine Translationcited by 139
  8. 2017Data Noising as Smoothing in Neural Network Language Modelscited by 78
  9. 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 75
  10. 2018SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translationcited by 72
  11. 2020Explicit Alignment Objectives for Multilingual Bidirectional Encoderscited by 50
  12. 2023When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languagescited by 32
  13. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 29
  14. 2024Code-Switching Curriculum Learning for Multilingual Transfer in LLMscited by 27
  15. 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 22
  16. 2024PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignmentcited by 20
  17. 2024Task-Adaptive Pretrained Language Models via Clustered-Importance Samplingcited by 18
  18. 2025Investigating and Scaling up Code-Switching for Multilingual Language Model Pre-Trainingcited by 16
  19. 2024Recent Advancements and Challenges of Turkic Central Asian Language Processingcited by 6
  20. 2024Training Bilingual LMs with Data Constraints in the Targeted Languagecited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review
  2. 2026Multilingual Amnesia: On the Transferability of Unlearning in Multilingual LLMs
  3. 2026Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM Unlearning
  4. 2023Translate to Disambiguate: Zero-shot Multilingual Word Sense Disambiguation with Pretrained Language Models
  5. 2022FreeTransfer-X: Safe and Label-Free Cross-Lingual Transfer from Off-the-Shelf Models
  6. 2022You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models
  7. 2025A Post-trainer's Guide to Multilingual Training Data: Uncovering Cross-lingual Transfer Dynamics
  8. 2025Edit Once, Update Everywhere: A Simple Framework for Cross-Lingual Knowledge Synchronization in LLMs