mlsanityData to 7 Oct 2026

Cross-Lingual Supervision improves Large Language Models Pre-training

arXiv 2305.11778 ↗First posted 19 May 2023cs.CL cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

The recent rapid progress in pre-training Large Language Models has relied on using self-supervised language modeling objectives like next token prediction or span corruption. On the other hand, Machine Translation Systems are mostly trained using cross-lingual supervision that requires aligned data between source and target languages. We demonstrate that pre-training Large Language Models on a mixture of a self-supervised Language Modeling objective and the supervised Machine Translation objective, therefore including cross-lingual parallel data during pre-training, yields models with better in-context learning abilities. As pre-training is a very resource-intensive process and a grid search on the best mixing ratio between the two objectives is prohibitively expensive, we propose a simple yet effective strategy to learn it during pre-training.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 18 of its 23 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

18 of 23 references.

References · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  5. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  6. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  7. 2022UL2: Unifying Language Learning Paradigmscited by 292
  8. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  9. 2017Automated Curriculum Learning for Neural Networkscited by 251
  10. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  11. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  12. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 144
6 more
  1. 2019The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-Englishcited by 61
  2. 2021MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairscited by 41
  3. 2022GEMv2: Multilingual NLG Benchmarking in a Single Line of Codecited by 36
  4. 2021nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?cited by 21
  5. 2021PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretrainingcited by 21
  6. 2021Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Banditscited by 16

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 7.

8 papers.

  1. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  2. 2024Fine-tuning Large Language Models for Domain-specific Machine Translationcited by 43
  3. 2023Fine-tuning Large Language Models for Adaptive Machine Translationcited by 21
  4. 2024Prefix Text as a Yarn: Eliciting Non-English Alignment in Foundation Language Modelcited by 10
  5. 2024Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Modelscited by 9
  6. 2024Enhancing Translation Accuracy of Large Language Models through Continual Pre-Training on Parallel Datacited by 7
  7. 2024Language Modelling Approaches to Adaptive Machine Translationcited by 3
  8. 2024ACT-MNMT Auto-Constriction Turning for Multilingual Neural Machine Translationcited by 0