mlsanityData to 14 Oct 2026

A Primer on Pretrained Multilingual Language Models

arXiv 2107.00676 ↗First posted 1 Jul 2021cs.CLcited by 49

Not on the map. Why?

Abstract (arXiv)

Multilingual Language Models (MLLMs) such as mBERT, XLM, XLM-R, textitetc. have emerged as a viable option for bringing the power of pretraining to a large number of languages. Given their success in zero-shot transfer learning, there has emerged a large body of work in (i) building bigger MLLMs~covering a large number of languages (ii) creating exhaustive benchmarks covering a wider variety of tasks and languages for evaluating MLLMs~ (iii) analysing the performance of MLLMs~on monolingual, zero-shot cross-lingual and bilingual tasks (iv) understanding the universal language patterns (if any) learnt by MLLMs~ and (v) augmenting the (often) limited capacity of MLLMs~ to improve their performance on seen or even unseen languages. In this survey, we review the existing literature covering the above broad areas of research pertaining to MLLMs. Based on our survey, we recommend some promising directions of future research.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Builds on 109 of its 147 references that are in the map. First posted on arXiv 1 Jul 2021.

Builds on

109 of 147 references.

References · 109

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  10. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
38 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  4. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  5. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  6. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  7. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  8. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  9. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  10. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  11. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  12. 2020Efficient Transformers: A Surveycited by 790
  13. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  14. 2018Computational Optimal Transportcited by 780
  15. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  16. 2019How multilingual is Multilingual BERT?cited by 645
  17. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  18. 2015Generating Sentences from a Continuous Spacecited by 612
  19. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  20. 2017Learning multiple visual domains with residual adapterscited by 557
  21. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  22. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  23. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  24. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
  25. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  26. 2019CamemBERT: a Tasty French Language Modelcited by 368
  27. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  28. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  29. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  30. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  31. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  32. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  33. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 222
  34. 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
  35. 2013Exploiting Similarities among Languages for Machine Translationcited by 191
  36. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  37. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  38. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 181

50 most cited of 109 All 109 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 9; 2023: 11; 2024: 10; 2025: 8; 2026: 8.

49 papers.

Sort:

By area:

  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  2. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 184
  3. 2024How do Large Language Models Handle Multilingualism?cited by 150
  4. 2022mGPT: Few-Shot Learners Go Multilingualcited by 142
  5. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 102
  6. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 84
  7. 2023Testing the Predictions of Surprisal Theory in 11 Languagescited by 77
  8. 2023Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Reviewcited by 50
  9. 2021When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfercited by 42
  10. 2023Revisiting Machine Translation for Cross-lingual Classificationcited by 41
  11. 2022SERENGETI: Massively Multilingual Language Models for Africacited by 40
  12. 2022Pretraining with Artificial Language: Studying Transferable Knowledge in Language Modelscited by 28
37 more
  1. 2024Privacy Preserving Prompt Engineering: A Surveycited by 25
  2. 2022Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Trainingcited by 24
  3. 2025LangTime: A Language-Guided Unified Model for Time Series Forecasting with Proximal Policy Optimizationcited by 22
  4. 2021mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Modelscited by 19
  5. 2023L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERTcited by 15
  6. 2023How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuningcited by 14
  7. 2024Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?cited by 11
  8. 2024Multilingual Brain Surgeon: Large Language Models Can be Compressed Leaving No Language Behindcited by 11
  9. 2023Exploring Large Language Models for Classical Philologycited by 10
  10. 2022CALCS 2021 Shared Task: Machine Translation for Code-Switched Datacited by 9
  11. 2022Linguistically inspired roadmap for building biologically reliable protein language modelscited by 7
  12. 2022The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfercited by 5
  13. 2025How Programming Concepts and Neurons Are Shared in Code Language Modelscited by 5
  14. 2025Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languagescited by 5
  15. 2026Bringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Modelscited by 5
  16. 2022Multilingual Transformer Encoders: a Word-Level Task-Agnostic Evaluationcited by 4
  17. 2023How Different Is Stereotypical Bias Across Languages?cited by 4
  18. 2024Understanding the role of FFNs in driving multilingual behaviour in LLMscited by 3
  19. 2024The Model Arena for Cross-lingual Sentiment Analysis: A Comparative Study in the Era of Large Language Modelscited by 3
  20. 2023Multilingual Multiword Expression Identification Using Lateral Inhibition and Domain Adaptationcited by 2
  21. 2023The Less the Merrier? Investigating Language Representation in Multilingual Modelscited by 2
  22. 2026Scaling Domain Data Repetition in LLM Pretrainingcited by 2
  23. 2023Investigating Lexical Sharing in Multilingual Machine Translation for Indian Languagescited by 1
  24. 2024Mitigating Language-Level Performance Disparity in mPLMs via Teacher Language Selection and Cross-lingual Self-Distillationcited by 1
  25. 2025Better To Ask in English? Evaluating Factual Accuracy of Multilingual LLMs in English and Low-Resource Languagescited by 1
  26. 2026Capacity-Aware Mixture Law Enables Efficient LLM Data Optimizationcited by 1
  27. 2026Multilinguality at the Edge: Developing Language Models for the Global Southcited by 1
  28. 2023Evaluating the Effectiveness of Pre-trained Language Models in Predicting the Helpfulness of Online Product Reviewscited by 0
  29. 2024Towards a World-English Language Model for On-Device Virtual Assistantscited by 0
  30. 2025Can MLLMs Generalize to Multi-Party dialog? Exploring Multilingual Response Generation in Complex Scenarioscited by 0
  31. 2025Duluth at SemEval-2025 Task 7: TF-IDF with Optimized Vector Dimensions for Multilingual Fact-Checked Claim Retrievalcited by 0
  32. 2025OpenEthics: A Comprehensive Ethical Evaluation of Open-Source Generative Large Language Modelscited by 0
  33. 2025A comparison of pipelines for the translation of a low resource language based on transformerscited by 0
  34. 2026A Study of Crosslinguistic Influence in Language Modelscited by 0
  35. 2026Effective vocabulary expansion of multilingual language models for extremely low-resource languagescited by 0
  36. 2026Lost in the Tower of Babel: The Adverse Effects of Incidental Multilingualism in LLMscited by 0
  37. 2026MultiGhostBench: A Multilingual Benchmark for Long-Form LLM-Generated Text Attribution under Distribution Shiftscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Cross-Lingual Text Classification with Multilingual Distillation and Zero-Shot-Aware Training
  2. 2024Multilingual Large Language Models: A Systematic Survey
  3. 2024Pruning Multilingual Large Language Models for Multilingual Inference
  4. 2024What Drives Performance in Multilingual Language Models?
  5. 2024Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean
  6. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers
  7. 2025Multilingual Language Model Pretraining using Machine-translated Data
  8. 2024Responsible Multilingual Large Language Models: A Survey of Development, Applications, and Societal Impact