mlsanityData to 7 Oct 2026

An Open Dataset and Model for Language Identification

arXiv 2305.13820 ↗First posted 23 May 2023cs.CLcited by 49

Not on the map. Why?

Abstract (arXiv)

Language identification (LID) is a fundamental step in many natural language processing pipelines. However, current LID systems are far from perfect, particularly on lower-resource languages. We present a LID model which achieves a macro-average F1 score of 0.93 and a false positive rate of 0.033 across 201 languages, outperforming previous work. We achieve this by training on a curated dataset of monolingual data, the reliability of which we ensure by auditing a sample from each source and each language manually. We make both the model and the dataset available to the research community. Finally, we carry out detailed analysis into our model's performance, both in comparison to existing open models and by language class.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 17 of its 55 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

17 of 55 references.

References · 17

  1. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  2. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  3. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  4. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  5. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  6. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  7. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  8. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 125
  9. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 76
  10. 2017The IIT Bombay English-Hindi Parallel Corpuscited by 74
  11. 2020Mapping Languages: The Corpus of Global Language Usecited by 42
  12. 2018Automatic Language Identification in Texts: A Surveycited by 40
5 more
  1. 2022AfroLID: A Neural Language Identification Tool for African Languagescited by 38
  2. 2021Many-to-English Machine Translation Tools, Data, and Pretrained Modelscited by 31
  3. 2021A Large-Scale Study of Machine Translation in the Turkic Languagescited by 21
  4. 2018MIZAN: A Large Persian-English Parallel Corpuscited by 13
  5. 2018The WiLI benchmark dataset for written language identificationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 14; 2025: 16; 2026: 14.

49 papers.

Sort:

By area:

  1. 2023Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpacacited by 60
  2. 2023Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decodingcited by 42
  3. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  4. 2023Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?cited by 39
  5. 2023GlotLID: Language Identification for Low-Resource Languagescited by 37
  6. 2024The Hidden Space of Transformer Language Adapterscited by 16
  7. 2024GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languagescited by 15
  8. 2025An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)cited by 15
  9. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 15
  10. 2025Multilingual Retrieval-Augmented Generation for Knowledge-Intensive Taskcited by 12
  11. 2025Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoderscited by 12
  12. 2024The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resourcescited by 11
37 more
  1. 2025AFRIDOC-MT: Document-level MT Corpus for African Languagescited by 11
  2. 2024MaskLID: Code-Switching Language Identification through Iterative Maskingcited by 10
  3. 2024mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpuscited by 8
  4. 2024The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insightscited by 7
  5. 2024LuxEmbedder: A Cross-Lingual Approach to Enhanced Luxembourgish Sentence Embeddingscited by 7
  6. 2025DocHPLT: A Massively Multilingual Document-Level Translation Datasetcited by 7
  7. 2023Fumbling in Babel: An Investigation into ChatGPT's Language Identification Abilitycited by 6
  8. 2024Geographically-Informed Language Identificationcited by 6
  9. 2024Chasing COMET: Leveraging Minimum Bayes Risk Decoding for Self-Improving Machine Translationcited by 6
  10. 2025Register Always Matters: Analysis of LLM Pretraining Data Through the Lens of Language Variationcited by 6
  11. 2025HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Modelscited by 6
  12. 2025ConLID: Supervised Contrastive Learning for Low-Resource Language Identificationcited by 5
  13. 2026CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Datacited by 5
  14. 2025Improving Multilingual Retrieval-Augmented Language Models through Dialectic Reasoning Argumentationscited by 4
  15. 2026OpenLID-v3: Improving the Precision of Closely Related Language Identification -- An Experience Reportcited by 4
  16. 2024Code-Switched Language Identification is Harder Than You Thinkcited by 3
  17. 2025Language Fusion for Parameter-Efficient Cross-lingual Transfercited by 3
  18. 2025DIVERS-Bench: Evaluating Language Identification Across Domain Shifts and Code-Switchingcited by 3
  19. 2024From N-grams to Pre-trained Multilingual Models For Language Identificationcited by 2
  20. 2025Multi-label Scandinavian Language Identification (SLIDE)cited by 2
  21. 2024FastSpell: the LangId Magic Spellcited by 1
  22. 2024FAME-MT Dataset: Formality Awareness Made Easy for Machine Translation Purposescited by 1
  23. 2025Improving Informally Romanized Language Identificationcited by 1
  24. 2025SciLaD: A Large-Scale, Transparent, Reproducible Dataset for Natural Scientific Language Processingcited by 1
  25. 2026What Language is This? Ask Your Tokenizercited by 1
  26. 2026LuxBorrow: From Pompier to Pompjee, Tracing Borrowing in Luxembourgishcited by 1
  27. 2026GaelEval: Benchmarking LLM Performance for Scottish Gaeliccited by 1
  28. 2026SomaliWeb v1: A Quality-Filtered Somali Web Corpus with a Matched Tokenizer and a Public Language-Identification Benchmarkcited by 1
  29. 2026Is a Document Educational or Just Wikipedia-Style? -- Pitfalls of Classifier-Based Quality Filteringcited by 1
  30. 2025Kr'eyoLID From Language Identification Towards Language Miningcited by 0
  31. 2026AfroScope: A Framework for Studying the Linguistic Landscape of Africacited by 0
  32. 2026How Should We Model the Probability of a Language?cited by 0
  33. 2026ltzGLUE: Luxembourgish General Language Understanding Evaluationcited by 0
  34. 2026Reference-Free Reinforcement Learning Fine-Tuning for MT: A Seq2Seq Perspectivecited by 0
  35. 2026CHALIS: A Challenge Dataset for Language Identification in Difficult Scenarioscited by 0
  36. 2026Loanword or Switch? The Annotation Boundary, Not the Model, Drives Kazakh-Russian Code-Switching Identificationcited by 0
  37. 2026Improving Language Identification for Code-Switched Utterances with Integer Linear Programmingcited by 0