mlsanityData to 13 Oct 2026

Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capability

arXiv 2305.10266 ↗First posted 17 May 2023cs.CLcited by 62

Not on the map. Why?

Abstract (arXiv)

Large, multilingual language models exhibit surprisingly good zero- or few-shot machine translation capabilities, despite having never seen the intentionally-included translation examples provided to typical neural translation systems. We investigate the role of incidental bilingualism -- the unintentional consumption of bilingual signals, including translation examples -- in explaining the translation capabilities of large language models, taking the Pathways Language Model (PaLM) as a case study. We introduce a mixed-method approach to measure and understand incidental bilingualism at scale. We show that PaLM is exposed to over 30 million translation pairs across at least 44 languages. Furthermore, the amount of incidental bilingual content is highly correlated with the amount of monolingual in-language content for non-English languages. We relate incidental bilingual content to zero-shot prompts and show that it can be used to mine new prompts to improve PaLM's out-of-English zero-shot translation quality. Finally, in a series of small-scale ablations, we show that its presence has a substantial impact on translation capabilities, although this impact diminishes with model scale.

Key facts. Cited by 62 papers in the map (data through 13 Oct 2026). Builds on 19 of its 39 references that are in the map. First posted on arXiv 17 May 2023.

Builds on

19 of 39 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,979
  7. 2019Language Models as Knowledge Bases?cited by 1,931
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,793
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,468
  10. 2019How Can We Know What Language Models Know?cited by 958
  11. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 705
  12. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 605
7 more
  1. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  2. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 485
  3. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 144
  4. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 92
  5. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 77
  6. 2018A Fast, Compact, Accurate Model for Language Identification of Codemixed Textcited by 11
  7. 2017Natural Language Processing with Small Feed-Forward Networkscited by 10

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 10; 2024: 29; 2025: 14; 2026: 9.

62 papers.

Sort:

By area:

  1. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  2. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  3. 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 137
  4. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  5. 2023Task Contamination: Language Models May Not Be Few-Shot Anymorecited by 99
  6. 2024On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neuronscited by 83
  7. 2025T*: Re-thinking Temporal Search for Long-Form Video Understandingcited by 82
  8. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  9. 2023Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpacacited by 60
  10. 2024CroissantLLM: A Truly Bilingual French-English Language Modelcited by 52
  11. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  12. 2024Crafting Personalized Agents through Retrieval-Augmented Generation on Editable Memory Graphscited by 38
38 more
  1. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Modelscited by 31
  2. 2023Do GPTs Produce Less Literal Translations?cited by 30
  3. 2025EuroBERT: Scaling Multilingual Encoders for European Languagescited by 29
  4. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 28
  5. 2023Iterative Translation Refinement with Large Language Modelscited by 27
  6. 2024Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowlycited by 27
  7. 2024Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Modelscited by 23
  8. 2024Salute the Classic: Revisiting Challenges of Machine Translation in the Age of Large Language Modelscited by 17
  9. 2024Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice?cited by 16
  10. 2025Investigating and Scaling up Code-Switching for Multilingual Language Model Pre-Trainingcited by 16
  11. 2025An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)cited by 15
  12. 2024Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translationcited by 13
  13. 2025Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translationcited by 13
  14. 2024Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?cited by 11
  15. 2024Dialectical Alignment: Resolving the Tension of 3H and Security Threats of LLMscited by 11
  16. 2025The Multilingual Divide and Its Impact on Global AI Safetycited by 10
  17. 2024An Empirical Study of In-context Learning in LLMs for Machine Translationcited by 9
  18. 2024The Role of Language Imbalance in Cross-lingual Generalisation: Insights from Cloned Language Experimentscited by 8
  19. 2024Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translationcited by 8
  20. 2025Just Go Parallel: Improving the Multilingual Capabilities of Large Language Modelscited by 8
  21. 2023Developing a Named Entity Recognition Dataset for Tagalogcited by 7
  22. 2024Enhancing Translation Accuracy of Large Language Models through Continual Pre-Training on Parallel Datacited by 7
  23. 2024How Multilingual Are Large Language Models Fine-Tuned for Translation?cited by 6
  24. 2025Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Expertscited by 6
  25. 2023On-the-Fly Fusion of Large Language Models and Machine Translationcited by 5
  26. 2024On the Evaluation Practices in Multilingual NLP: Can Machine Translation Offer an Alternative to Human Translations?cited by 5
  27. 2024Principled Understanding of Generalization for Generative Transformer Models in Arithmetic Reasoning Taskscited by 5
  28. 2024DEPT: Decoupled Embeddings for Pre-training Language Modelscited by 5
  29. 2024Questionable practices in machine learningcited by 4
  30. 2024Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMscited by 4
  31. 2023SCALE: Synergized Collaboration of Asymmetric Language Translation Enginescited by 3
  32. 2024Self-Translate-Train: Enhancing Cross-Lingual Transfer of Large Language Models via Inherent Capabilitycited by 3
  33. 2026OpenSeal: Good, Fast, and Cheap Construction of an Open-Source Southeast Asian LLM via Parallel Datacited by 3
  34. 2023When Does Monolingual Data Help Multilingual Translation: The Role of Domain and Model Scalecited by 2
  35. 2023On Bilingual Lexicon Induction with Large Language Modelscited by 2
  36. 2025Translation in the Wildcited by 2
  37. 2025Prompting LLMs: Length Control for Isometric Machine Translationcited by 2
  38. 2025How Important is `Perfect' English for Machine Translation Prompts?cited by 2

50 most cited of 62 All 62 (JSON)