mlsanityData to 7 Oct 2026

GlotLID: Language Identification for Low-Resource Languages

arXiv 2310.16248 ↗First posted 24 Oct 2023cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

Several recent papers have published good solutions for language identification (LID) for about 300 high-resource and medium-resource languages. However, there is no LID available that (i) covers a wide range of low-resource languages, (ii) is rigorously evaluated and reliable and (iii) efficient and easy to use. Here, we publish GlotLID-M, an LID model that satisfies the desiderata of wide coverage, reliability and efficiency. It identifies 1665 languages, a large increase in coverage compared to prior work. In our experiments, GlotLID-M outperforms four baselines (CLD3, FT176, OpenLID and NLLB) when balancing F1 and false positive rate (FPR). We analyze the unique challenges that low-resource LID poses: incorrect corpus metadata, leakage from high-resource languages, difficulty separating closely related languages, handling of macrolanguage vs varieties and in general noisy data. We hope that integrating GlotLID-M into dataset creation pipelines will improve quality and enhance accessibility of NLP technology for low-resource languages and cultures. GlotLID-M model (including future versions), code, and list of data sources are available: https://github.com/cisnlp/GlotLID.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 35 of its 85 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

35 of 85 references.

References · 35

  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  3. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  4. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  5. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  6. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  7. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  8. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  9. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 269
  10. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  11. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  12. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 125
23 more
  1. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  2. 2020LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluationcited by 92
  3. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 90
  4. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 76
  5. 2017The IIT Bombay English-Hindi Parallel Corpuscited by 74
  6. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  7. 2023An Open Dataset and Model for Language Identificationcited by 49
  8. 2022Towards Afrocentric NLP for African Languages: Where We Are and Where We Can Gocited by 46
  9. 2020Mapping Languages: The Corpus of Global Language Usecited by 42
  10. 2018Automatic Language Identification in Texts: A Surveycited by 40
  11. 2022AfroLID: A Neural Language Identification Tool for African Languagescited by 38
  12. 2021Many-to-English Machine Translation Tools, Data, and Pretrained Modelscited by 31
  13. 2021A Large-Scale Study of Machine Translation in the Turkic Languagescited by 21
  14. 2021AI4D -- African Language Programcited by 18
  15. 2023A Survey of Corpora for Germanic Low-Resource Languages and Dialectscited by 16
  16. 2023GlotScript: A Resource and Tool for Low Resource Writing System Identificationcited by 15
  17. 2018MIZAN: A Large Persian-English Parallel Corpuscited by 13
  18. 2023Preparing the Vuk'uzenzele and ZA-gov-multilingual South African multilingual corporacited by 12
  19. 2017LanideNN: Multilingual Language Identification on Character Windowcited by 10
  20. 2017Natural Language Processing with Small Feed-Forward Networkscited by 10
  21. 2020Discriminating Between Similar Nordic Languagescited by 10
  22. 2018The WiLI benchmark dataset for written language identificationcited by 8
  23. 2023PALI: A Language Identification Benchmark for Perso-Arabic Scriptscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 15; 2026: 13.

37 papers.

Sort:

By area:

  1. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 78
  2. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  3. 2025IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistancecited by 22
  4. 2024GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languagescited by 15
  5. 2024MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructionscited by 12
  6. 2024Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?cited by 11
  7. 2024Why do LLaVA Vision-Language Models Reply to Images in English?cited by 11
  8. 2025Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resourcescited by 10
  9. 2023CreoleVal: Multilingual Multitask Benchmarks for Creolescited by 9
  10. 2025M-ABSA: A Multilingual Dataset for Aspect-Based Sentiment Analysiscited by 9
  11. 2025BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Datacited by 9
  12. 2025NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmarkcited by 7
25 more
  1. 2025mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Taskscited by 7
  2. 2024Lens: Rethinking Multilingual Enhancement for Large Language Modelscited by 5
  3. 2025DCAD-2000: A Multilingual Dataset across 2000+ Languages with Data Cleaning as Anomaly Detectioncited by 5
  4. 2025Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Datacited by 4
  5. 2025Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updatescited by 4
  6. 2025DIVERS-Bench: Evaluating Language Identification Across Domain Shifts and Code-Switchingcited by 3
  7. 2026Mending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual Translationcited by 3
  8. 2025Revisiting Direct Speech-to-Text Translation with Speech LLMs: Better Scaling than CoT Prompting?cited by 2
  9. 2026Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domaincited by 2
  10. 2026GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scriptscited by 2
  11. 2025MUG-Eval: A Proxy Evaluation Framework for Multilingual Generation Capabilities in Any Languagecited by 1
  12. 2025TalTech Systems for the Interspeech 2025 ML-SUPERB 2.0 Challengecited by 1
  13. 2026What Language is This? Ask Your Tokenizercited by 1
  14. 2026SomaliWeb v1: A Quality-Filtered Somali Web Corpus with a Matched Tokenizer and a Public Language-Identification Benchmarkcited by 1
  15. 2026Laguna M.1/XS.2 Technical Reportcited by 1
  16. 2026The Fragility of Chain-of-Thought Monitoring Across Typologically Diverse Languagescited by 1
  17. 2026Auditing Cross-Lingual Fairness in Language Model Watermarkingcited by 1
  18. 2024NusaMT-7B: Machine Translation for Low-Resource Indonesian Languages with Large Language Modelscited by 0
  19. 2025Bias beyond Borders: Global Inequalities in AI-Generated Musiccited by 0
  20. 2025Language Diversity: Evaluating Language Usage and AI Performance on African Languages in Digital Spacescited by 0
  21. 2026Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindicited by 0
  22. 2026Lost in the Tower of Babel: The Adverse Effects of Incidental Multilingualism in LLMscited by 0
  23. 2026KletterMix: Climbing Toward High-Quality German Pretraining Data - The Full Reportcited by 0
  24. 2026Enhancing Low-Resource Language Reasoning via High-Resource Language Feature Transfercited by 0
  25. 2026Evaluating Communicative Success in Machine-Translated Conversationcited by 0