mlsanityData to 7 Oct 2026

OpenLID-v3: Improving the Precision of Closely Related Language Identification -- An Experience Report

arXiv 2602.13139 ↗First posted 13 Feb 2026cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Language identification (LID) is an essential step in building high-quality multilingual datasets from web data. Existing LID tools (such as OpenLID or GlotLID) often struggle to identify closely related languages and to distinguish valid natural language from noise, which contaminates language-specific subsets, especially for low-resource languages. In this work we extend the OpenLID classifier by adding more training data, merging problematic language variant clusters, and introducing a special label for marking noise. We call this extended system OpenLID-v3 and evaluate it against GlotLID on multiple benchmarks. During development, we focus on three groups of closely related languages (Bosnian, Croatian, and Serbian; Romance varieties of Northern Italy and Southern France; and Scandinavian languages) and contribute new evaluation datasets where existing ones are inadequate. We find that ensemble approaches improve precision but also substantially reduce coverage for low-resource languages. OpenLID-v3 is available on https://huggingface.co/HPLT/OpenLID-v3.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 27 of its 90 references that are in the map. First posted on arXiv 13 Feb 2026.

Builds on

27 of 90 references.

References · 27

  1. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  2. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  3. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  4. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  5. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 269
  6. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  7. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 134
  8. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 125
  9. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 76
  10. 2017The IIT Bombay English-Hindi Parallel Corpuscited by 74
  11. 2022NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languagescited by 67
  12. 2024A New Massive Multilingual Dataset for High-Performance Language Technologiescited by 56
15 more
  1. 2023An Open Dataset and Model for Language Identificationcited by 49
  2. 2018Automatic Language Identification in Texts: A Surveycited by 40
  3. 2020PMIndia -- A Collection of Parallel Corpora of Languages of Indiacited by 40
  4. 2022AfroLID: A Neural Language Identification Tool for African Languagescited by 38
  5. 2023AfriQA: Cross-lingual Open-Retrieval Question Answering for African Languagescited by 23
  6. 2021A Large-Scale Study of Machine Translation in the Turkic Languagescited by 21
  7. 2023Findings of the VarDial Evaluation Campaign 2023cited by 14
  8. 2018MIZAN: A Large Persian-English Parallel Corpuscited by 13
  9. 2023Preparing the Vuk'uzenzele and ZA-gov-multilingual South African multilingual corporacited by 12
  10. 2017LanideNN: Multilingual Language Identification on Character Windowcited by 10
  11. 2020Discriminating Between Similar Nordic Languagescited by 10
  12. 2018The WiLI benchmark dataset for written language identificationcited by 8
  13. 2023LIMIT: Language Identification, Misidentification, and Translation using Hierarchical Models in 350+ Languagescited by 6
  14. 2026CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Datacited by 5
  15. 2025Multi-label Scandinavian Language Identification (SLIDE)cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modelingcited by 1
  2. 2026Why Low-Resource NLP Needs More Than Cross-Lingual Transfer: Lessons Learned from Luxembourgishcited by 0
  3. 2026On the Limits of Model Merging for Multilinguality in Pre-Trainingcited by 0
  4. 2026Reference-Free Post-Training of Open Large Language Models for Multilingual Machine Translationcited by 0