mlsanityData to 14 Oct 2026

Soft Language Clustering for Multilingual Model Pre-training

arXiv 2306.07610 ↗First posted 13 Jun 2023cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Multilingual pre-trained language models have demonstrated impressive (zero-shot) cross-lingual transfer abilities, however, their performance is hindered when the target language has distant typology from source languages or when pre-training data is limited in size. In this paper, we propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods. On the tasks of XTREME including text classification, sequence labeling, question answering, and sentence retrieval, both base- and large-size language models pre-trained with our proposed method exhibit consistent performance improvement. Furthermore, it provides substantial advantages for low-resource languages in unsupervised sentence retrieval and for target languages that differ greatly from the source language in cross-lingual transfer.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

34 of 46 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  12. 2019Cross-lingual Language Model Pretrainingcited by 1,090
22 more
  1. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  2. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  3. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  4. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  5. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  6. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  7. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  8. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  9. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  10. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  11. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  12. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  13. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  14. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  15. 2020ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corporacited by 63
  16. 2021Discrete and Soft Prompting for Multilingual Modelscited by 49
  17. 2021Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignmentcited by 41
  18. 2023Language Models are Few-shot Learners for Prognostic Predictioncited by 37
  19. 2020VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generationcited by 34
  20. 2020Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translationcited by 28
  21. 2022Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Promptcited by 25
  22. 2022Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfercited by 22

Built on by

Cited by year

Citing papers per year: 2024: 4.

4 papers.

  1. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  2. 2024The Zeno's Paradox of `Low-Resource' Languagescited by 12
  3. 2024Measuring Cross-lingual Transfer in Bytescited by 6
  4. 2024Responsible Multilingual Large Language Models: A Survey of Development, Applications, and Societal Impactcited by 3