mlsanityData to 14 Oct 2026

Larger-Scale Transformers for Multilingual Masked Language Modeling

arXiv 2105.00572 ↗First posted 2 May 2021cs.CLcited by 87

Not on the map. Why?

Abstract (arXiv)

Recent work has demonstrated the effectiveness of cross-lingual language model pretraining for cross-lingual understanding. In this study, we present the results of two larger multilingual masked language models, with 3.5B and 10.7B parameters. Our two new models dubbed XLM-R XL and XLM-R XXL outperform XLM-R by 1.8% and 2.4% average accuracy on XNLI. Our model also outperforms the RoBERTa-Large model on several English tasks of the GLUE benchmark by 0.3% on average while handling 99 more languages. This suggests pretrained models with larger capacity may obtain both strong performance on high-resource languages while greatly improving low-resource languages. We make our code and models publicly available.

Key facts. Cited by 87 papers in the map (data through 14 Oct 2026). Builds on 21 of its 22 references that are in the map. First posted on arXiv 2 May 2021.

Builds on

21 of 22 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
9 more
  1. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  2. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  3. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  4. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  5. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  6. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  7. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  8. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  9. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 257

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 8; 2023: 28; 2024: 31; 2025: 8; 2026: 5.

87 papers.

Sort:

By area:

  1. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  2. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  3. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  4. 2024Scaling up Masked Diffusion Models on Textcited by 199
  5. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 177
  6. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 151
  7. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 134
  8. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  9. 2023Intrinsic Dimension Estimation for Robust Detection of AI-Generated Textscited by 112
  10. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  11. 2021Language Models are Few-shot Multilingual Learnerscited by 92
  12. 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
38 more
  1. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  2. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 84
  3. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  4. 2022SERENGETI: Massively Multilingual Language Models for Africacited by 40
  5. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuningcited by 39
  6. 2024Breaking the Curse of Multilinguality with Cross-lingual Expert Language Modelscited by 38
  7. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  8. 2024mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generationcited by 32
  9. 2025EuroBERT: Scaling Multilingual Encoders for European Languagescited by 31
  10. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 29
  11. 2023Efficiently Adapting Pretrained Language Models To New Languagescited by 28
  12. 2021Server-Side Local Gradient Averaging and Learning Rate Acceleration for Scalable Split Learningcited by 23
  13. 2022Causes and Cures for Interference in Multilingual Translationcited by 23
  14. 2023An Efficient Multilingual Language Model Compression through Vocabulary Trimmingcited by 20
  15. 2022pNLP-Mixer: an Efficient all-MLP Architecture for Languagecited by 19
  16. 2023TaCo: Enhancing Cross-Lingual Transfer for Low-Resource Languages in LLMs through Translation-Assisted Chain-of-Thought Processescited by 19
  17. 2023LMCap: Few-shot Multilingual Image Captioning by Retrieval Augmented Language Model Promptingcited by 17
  18. 2023Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Expertscited by 17
  19. 2022Beyond English-Centric Bitexts for Better Multilingual Language Representation Learningcited by 16
  20. 2023Navigating Cultural Chasms: Exploring and Unlocking the Cultural POV of Text-To-Image Modelscited by 16
  21. 2023Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Modelscited by 16
  22. 2024QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translationcited by 16
  23. 2023MELA: Multilingual Evaluation of Linguistic Acceptabilitycited by 14
  24. 2023Bringing order into the realm of Transformer-based language models for artificial intelligence and lawcited by 13
  25. 2024Improving Minimum Bayes Risk Decoding with Multi-Promptcited by 13
  26. 2023SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domainscited by 12
  27. 2024BERTs are Generative In-Context Learnerscited by 12
  28. 2023SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Windowcited by 11
  29. 2024Multilingual Brain Surgeon: Large Language Models Can be Compressed Leaving No Language Behindcited by 11
  30. 2024Cross-Lingual Transfer Learning for Speech Translationcited by 11
  31. 2023Target-Agnostic Gender-Aware Contrastive Learning for Mitigating Bias in Multilingual Machine Translationcited by 10
  32. 2024LinguAlchemy: Fusing Typological and Geographical Elements for Unseen Language Generalizationcited by 10
  33. 2024Beyond Correlation: Interpretable Evaluation of Machine Translation Metricscited by 10
  34. 2022What Do Compressed Multilingual Machine Translation Models Forget?cited by 9
  35. 2023Bridging the Language Gap: Dynamic Learning Strategies for Improving Multilingual Performance in LLMscited by 8
  36. 2023Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosioncited by 8
  37. 2024UQA: Corpus for Urdu Question Answeringcited by 8
  38. 2021A cost-benefit analysis of cross-lingual transfer methodscited by 7

50 most cited of 87 All 87 (JSON)