mlsanityData to 7 Oct 2026

Cross-Lingual Language Model Meta-Pretraining

arXiv 2109.11129 ↗First posted 23 Sep 2021cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

The success of pretrained cross-lingual language models relies on two essential abilities, i.e., generalization ability for learning downstream tasks in a source language, and cross-lingual transferability for transferring the task knowledge to other languages. However, current methods jointly learn the two abilities in a single-phase cross-lingual pretraining process, resulting in a trade-off between generalization and cross-lingual transfer. In this paper, we propose cross-lingual language model meta-pretraining, which learns the two abilities in different training phases. Our method introduces an additional meta-pretraining phase before cross-lingual pretraining, where the model learns generalization ability on a large-scale monolingual corpus. Then, the model focuses on learning cross-lingual transfer on a multilingual corpus. Experimental results show that our method improves both generalization and cross-lingual transfer, and produces better-aligned representations across different languages.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 44 of its 51 references that are in the map. First posted on arXiv 23 Sep 2021.

Builds on

44 of 51 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
32 more
  1. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  2. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  3. 2019How multilingual is Multilingual BERT?cited by 642
  4. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  5. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  6. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  7. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  8. 2017Word Translation Without Parallel Datacited by 480
  9. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  10. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  11. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  12. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  13. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  14. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  15. 2020SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddingscited by 155
  16. 2020Multilingual Alignment of Contextual Word Representationscited by 113
  17. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  18. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  19. 2020Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Informationcited by 64
  20. 2019Cross-Lingual Natural Language Generation via Pre-Trainingcited by 63
  21. 2020On Learning Universal Representations Across Languagescited by 58
  22. 2020Explicit Alignment Objectives for Multilingual Bidirectional Encoderscited by 50
  23. 2021DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoderscited by 50
  24. 2020Inducing Language-Agnostic Multilingual Representationscited by 48
  25. 2021MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairscited by 41
  26. 2021Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignmentcited by 41
  27. 2021Consistency Regularization for Cross-Lingual Fine-Tuningcited by 41
  28. 2020VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generationcited by 34
  29. 2021ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generationcited by 23
  30. 2021nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?cited by 21
  31. 2021On the ability of monolingual models to learn language-agnostic representationscited by 5
  32. 2021Cross-lingual Transferring of Pre-trained Contextualized Language Modelscited by 3

Built on by

Not cited yet.