mlsanityData to 7 Oct 2026

Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Models

arXiv 2204.08110 ↗First posted 17 Apr 2022cs.CLcited by 90

Not on the map. Why?

Abstract (arXiv)

English pretrained language models, which make up the backbone of many modern NLP systems, require huge amounts of unlabeled training data. These models are generally presented as being trained only on English text but have been found to transfer surprisingly well to other languages. We investigate this phenomenon and find that common English pretraining corpora actually contain significant amounts of non-English text: even when less than 1% of data is not English (well within the error rate of strong language classifiers), this leads to hundreds of millions of foreign language tokens in large-scale datasets. We then demonstrate that even these small percentages of non-English data facilitate cross-lingual transfer for models trained on them, with target language performance strongly correlated to the amount of in-language data seen during pretraining. In light of these findings, we argue that no model is truly monolingual when pretrained at scale, which should be considered when evaluating cross-lingual transfer.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Builds on 22 of its 29 references that are in the map. First posted on arXiv 17 Apr 2022.

Builds on

22 of 29 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  8. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  10. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  11. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  12. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
10 more
  1. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 269
  2. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  3. 2018A Simple Method for Commonsense Reasoningcited by 223
  4. 2022Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selectioncited by 52
  5. 2017Learning Language Representations for Typology Predictioncited by 40
  6. 2020From English To Foreign Languages: Transferring Pre-trained Language Modelscited by 37
  7. 2021Cross-lingual Transfer of Monolingual Modelscited by 14
  8. 2019Can Monolingual Pretrained Models Help Cross-Lingual Classification?cited by 6
  9. 2021On the ability of monolingual models to learn language-agnostic representationscited by 5
  10. 2021Cross-lingual Transferring of Pre-trained Contextualized Language Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 28; 2024: 32; 2025: 15; 2026: 13.

90 papers.

Sort:

By area:

  1. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  2. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  3. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 136
  4. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 123
  5. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 117
  6. 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languagescited by 96
  7. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 92
  8. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 78
  9. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
  10. 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
  11. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  12. 2023Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capabilitycited by 62
38 more
  1. 2024Quantifying Contamination in Evaluating Code Generation Capabilities of Language Modelscited by 62
  2. 2023Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpacacited by 60
  3. 2023Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOMcited by 58
  4. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  5. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 42
  6. 2024Question Translation Training for Better Multilingual Reasoningcited by 41
  7. 2024MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modelingcited by 40
  8. 2022Prompting Language Models for Linguistic Structurecited by 39
  9. 2023Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?cited by 39
  10. 2025Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMscited by 39
  11. 2023GlotLID: Language Identification for Low-Resource Languagescited by 37
  12. 2024Breaking the Curse of Multilinguality with Cross-lingual Expert Language Modelscited by 36
  13. 2023Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrationscited by 34
  14. 2023On Evaluating and Mitigating Gender Biases in Multilingual Settingscited by 31
  15. 2024Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediationcited by 30
  16. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27
  17. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 27
  18. 2022GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Costcited by 24
  19. 2024Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Modelscited by 22
  20. 2025Lessons From Red Teaming 100 Generative AI Productscited by 22
  21. 2023Position: Key Claims in LLM Research Have a Long Tail of Footnotescited by 20
  22. 2023Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?cited by 19
  23. 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understandingcited by 17
  24. 2023Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Modelscited by 16
  25. 2024Probabilistic Medical Predictions of Large Language Modelscited by 16
  26. 2024GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languagescited by 15
  27. 2024PRESTO: Progressive Pretraining Enhances Synthetic Chemistry Outcomescited by 14
  28. 2025Understanding In-Context Machine Translation for Low-Resource Languages: A Case Study on Manchucited by 12
  29. 2024Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?cited by 11
  30. 2024Why do LLaVA Vision-Language Models Reply to Images in English?cited by 11
  31. 2024Beyond Math: Stories as a Testbed for Memorization-Constrained Reasoning in LLMscited by 11
  32. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 11
  33. 2023Large Language Models Leverage External Knowledge to Extend Clinical Insight Beyond Language Boundariescited by 10
  34. 2024Zero-shot cross-lingual transfer in instruction tuning of large language modelscited by 10
  35. 2024CrossIn: An Efficient Instruction Tuning Approach for Cross-Lingual Knowledge Alignmentcited by 10
  36. 2024Adaptive Pre-training Data Detection for Large Language Models via Surprising Tokenscited by 10
  37. 2025The Multilingual Divide and Its Impact on Global AI Safetycited by 10
  38. 2023Continual Learning Under Language Shiftcited by 9

50 most cited of 90 All 90 (JSON)