mlsanityData to 14 Oct 2026

Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages

arXiv 2305.12182 ↗First posted 20 May 2023cs.CLcited by 116

Not on the map. Why?

Abstract (arXiv)

The NLP community has mainly focused on scaling Large Language Models (LLMs) vertically, i.e., making them better for about 100 languages. We instead scale LLMs horizontally: we create, through continued pretraining, Glot500-m, an LLM that covers 511 predominantly low-resource languages. An important part of this effort is to collect and clean Glot500-c, a corpus that covers these 511 languages and allows us to train Glot500-m. We evaluate Glot500-m on five diverse tasks across these languages. We observe large improvements for both high-resource and low-resource languages compared to an XLM-R baseline. Our analysis shows that no single factor explains the quality of multilingual LLM representations. Rather, a combination of factors determines quality including corpus size, script, "help" from related languages and the total capacity of the model. Our work addresses an important goal of NLP research: we should not limit NLP to a small fraction of the world's languages and instead strive to support as many languages as possible to bring the benefits of NLP technology to all languages and cultures. Code, data and models are available at https://github.com/cisnlp/Glot500.

Key facts. Cited by 116 papers in the map (data through 14 Oct 2026). Builds on 50 of its 91 references that are in the map. First posted on arXiv 20 May 2023.

Builds on

50 of 91 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  9. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  10. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  11. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  12. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
38 more
  1. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  2. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  3. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  4. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  5. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  6. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 223
  7. 2020SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddingscited by 156
  8. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  9. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  10. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  11. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  12. 2022A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translationcited by 98
  13. 2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 93
  14. 2017The IIT Bombay English-Hindi Parallel Corpuscited by 75
  15. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  16. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  17. 2020TICO-19: the Translation Initiative for Covid-19cited by 68
  18. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  19. 2020Improving Multilingual Models with Language-Clustered Vocabulariescited by 53
  20. 2021Revisiting the Primacy of English in Zero-shot Cross-lingual Transfercited by 47
  21. 2020Arabic Dialect Identification in the Wildcited by 45
  22. 2022Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptationcited by 44
  23. 2020Mapping Languages: The Corpus of Global Language Usecited by 42
  24. 2022SERENGETI: Massively Multilingual Language Models for Africacited by 40
  25. 2021The Effect of Domain and Diacritics in Yor`ub'a-English Neural Machine Translationcited by 38
  26. 2021Many-to-English Machine Translation Tools, Data, and Pretrained Modelscited by 31
  27. 2019Improving Pre-Trained Multilingual Models with Vocabulary Expansioncited by 28
  28. 2022Bloom Library: Multimodal Datasets in 300+ Languages for a Variety of Downstream Taskscited by 28
  29. 2023Taxi1500: A Multilingual Dataset for Text Classification in 1500 Languagescited by 23
  30. 2019JParaCrawl: A Large Scale Web-Based English-Japanese Parallel Corpuscited by 21
  31. 2021A Large-Scale Study of Machine Translation in the Turkic Languagescited by 21
  32. 2022Towards the Next 1000 Languages in Multilingual Machine Translation: Exploring the Synergy Between Supervised and Self-Supervised Learningcited by 19
  33. 2020Towards Computational Linguistics in Minangkabau Language: Studies on Sentiment Analysis and Machine Translationcited by 16
  34. 2022Multilingual Open Text Release 1: Public Domain News in 44 Languagescited by 10
  35. 2023NLNDE at SemEval-2023 Task 12: Adaptive Pretraining and Source Language Selection for Low-Resource Multilingual Sentiment Analysiscited by 10
  36. 2022Towards a Broad Coverage Named Entity Resource: A Data-Efficient Approach for Many Diverse Languagescited by 8
  37. 2016A Large-Scale Multilingual Disambiguation of Glossescited by 5
  38. 2022Graph-Based Multilingual Label Propagation for Low-Resource Part-of-Speech Taggingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 18; 2024: 52; 2025: 32; 2026: 14.

116 papers.

Sort:

By area:

  1. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  2. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 181
  3. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 152
  4. 2024Towards Conversational Diagnostic AIcited by 79
  5. 2023MultiLegalPile: A 689GB Multilingual Legal Corpuscited by 60
  6. 2024A New Massive Multilingual Dataset for High-Performance Language Technologiescited by 56
  7. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  8. 2023GlotLID: Language Identification for Low-Resource Languagescited by 39
  9. 2023Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?cited by 39
  10. 2023OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretrainingcited by 37
  11. 2023Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrationscited by 35
  12. 2023When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languagescited by 32
38 more
  1. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 29
  2. 2024MaLA-500: Massive Language Adaptation of Large Language Modelscited by 28
  3. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27
  4. 2024Large Language Model (LLM) AI text generation detection based on transformer deep learning algorithmcited by 23
  5. 2023TaCo: Enhancing Cross-Lingual Transfer for Low-Resource Languages in LLMs through Translation-Assisted Chain-of-Thought Processescited by 19
  6. 2024The Hidden Space of Transformer Language Adapterscited by 17
  7. 2025Multilingual Language Model Pretraining using Machine-translated Datacited by 17
  8. 2024GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languagescited by 16
  9. 2025An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)cited by 16
  10. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 16
  11. 2023Translation and Fusion Improves Zero-shot Cross-lingual Information Extractioncited by 15
  12. 2023GlotScript: A Resource and Tool for Low Resource Writing System Identificationcited by 15
  13. 2024Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domaincited by 15
  14. 2024What Drives Performance in Multilingual Language Models?cited by 15
  15. 2024Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignmentcited by 14
  16. 2025Compositional Translation: A Novel LLM-based Approach for Low-resource Machine Translationcited by 14
  17. 2024Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translationcited by 13
  18. 2024"It's how you do things that matters": Attending to Process to Better Serve Indigenous Communities with Language Technologiescited by 12
  19. 2024Shallow Synthesis of Knowledge in GPT-Generated Texts: A Case Study in Automatic Related Work Compositioncited by 12
  20. 2024A Principled Framework for Evaluating on Typologically Diverse Languagescited by 12
  21. 2024MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructionscited by 12
  22. 2024Exploring Multilingual Probing in Large Language Models: A Cross-Language Analysiscited by 12
  23. 2024The Zeno's Paradox of `Low-Resource' Languagescited by 12
  24. 2024Small Languages, Big Models: A Study of Continual Training on Languages of Norwaycited by 12
  25. 2025SMOL: Professionally translated parallel data for 115 under-represented languagescited by 12
  26. 2023Crosslingual Transfer Learning for Low-Resource Languages Based on Multilingual Colexification Graphscited by 11
  27. 2024Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?cited by 11
  28. 2024MINERS: Multilingual Language Models as Semantic Retrieverscited by 11
  29. 2025AfriHate: A Multilingual Collection of Hate Speech and Abusive Language Datasets for African Languagescited by 11
  30. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 11
  31. 2025Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindicited by 10
  32. 2025Tracing Multilingual Factual Knowledge Acquisition in Pretrainingcited by 10
  33. 2024Natural Language Processing RELIES on Linguisticscited by 9
  34. 2024TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Datacited by 9
  35. 2024Targeted Multilingual Adaptation for Low-resource Language Familiescited by 9
  36. 2024Constrained Decoding for Cross-lingual Label Projectioncited by 8
  37. 2024Exploring the Role of Transliteration in In-Context Learning for Low-resource Languages Written in Non-Latin Scriptscited by 8
  38. 2025Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansioncited by 8

50 most cited of 116 All 116 (JSON)