# An Open Dataset and Model for Language Identification

arXiv: https://arxiv.org/abs/2305.13820 · HTML: https://mlsanity.com/paper/2305.13820/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 55 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 17 of 17)

- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?](https://mlsanity.com/paper/1804.06323/) (2018)
- [Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus](https://mlsanity.com/paper/2010.14571/) (2020)
- [The IIT Bombay English-Hindi Parallel Corpus](https://mlsanity.com/paper/1710.02855/) (2017)
- [Mapping Languages: The Corpus of Global Language Use](https://mlsanity.com/paper/2004.00798/) (2020)
- [Automatic Language Identification in Texts: A Survey](https://mlsanity.com/paper/1804.08186/) (2018)
- [AfroLID: A Neural Language Identification Tool for African Languages](https://mlsanity.com/paper/2210.11744/) (2022)
- [Many-to-English Machine Translation Tools, Data, and Pretrained Models](https://mlsanity.com/paper/2104.00290/) (2021)
- [A Large-Scale Study of Machine Translation in the Turkic Languages](https://mlsanity.com/paper/2109.04593/) (2021)
- [MIZAN: A Large Persian-English Parallel Corpus](https://mlsanity.com/paper/1801.02107/) (2018)
- [The WiLI benchmark dataset for written language identification](https://mlsanity.com/paper/1801.07779/) (2018)

## Built on by (most cited first, 30 of 50)

- [Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca](https://mlsanity.com/paper/2309.08958/) (2023)
- [Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decoding](https://mlsanity.com/paper/2309.07098/) (2023)
- [What is the Role of Small Models in the LLM Era: A Survey](https://mlsanity.com/paper/2409.06857/) (2024)
- [GlotLID: Language Identification for Low-Resource Languages](https://mlsanity.com/paper/2310.16248/) (2023)
- [Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?](https://mlsanity.com/paper/2312.12683/) (2023)
- [The Hidden Space of Transformer Language Adapters](https://mlsanity.com/paper/2402.13137/) (2024)
- [GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages](https://mlsanity.com/paper/2410.23825/) (2024)
- [An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)](https://mlsanity.com/paper/2503.10267/) (2025)
- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders](https://mlsanity.com/paper/2505.05111/) (2025)
- [Multilingual Retrieval-Augmented Generation for Knowledge-Intensive Task](https://mlsanity.com/paper/2504.03616/) (2025)
- [MaskLID: Code-Switching Language Identification through Iterative Masking](https://mlsanity.com/paper/2406.06263/) (2024)
- [The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources](https://mlsanity.com/paper/2406.16746/) (2024)
- [AFRIDOC-MT: Document-level MT Corpus for African Languages](https://mlsanity.com/paper/2501.06374/) (2025)
- [mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus](https://mlsanity.com/paper/2406.08707/) (2024)
- [The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insights](https://mlsanity.com/paper/2405.01345/) (2024)
- [LuxEmbedder: A Cross-Lingual Approach to Enhanced Luxembourgish Sentence Embeddings](https://mlsanity.com/paper/2412.03331/) (2024)
- [DocHPLT: A Massively Multilingual Document-Level Translation Dataset](https://mlsanity.com/paper/2508.13079/) (2025)
- [HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models](https://mlsanity.com/paper/2511.01066/) (2025)
- [Fumbling in Babel: An Investigation into ChatGPT's Language Identification Ability](https://mlsanity.com/paper/2311.09696/) (2023)
- [Geographically-Informed Language Identification](https://mlsanity.com/paper/2403.09892/) (2024)
- [Chasing COMET: Leveraging Minimum Bayes Risk Decoding for Self-Improving Machine Translation](https://mlsanity.com/paper/2405.11937/) (2024)
- [Register Always Matters: Analysis of LLM Pretraining Data Through the Lens of Language Variation](https://mlsanity.com/paper/2504.01542/) (2025)
- [ConLID: Supervised Contrastive Learning for Low-Resource Language Identification](https://mlsanity.com/paper/2506.15304/) (2025)
- [CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data](https://mlsanity.com/paper/2601.18026/) (2026)
- [Improving Multilingual Retrieval-Augmented Language Models through Dialectic Reasoning Argumentations](https://mlsanity.com/paper/2504.04771/) (2025)
- [OpenLID-v3: Improving the Precision of Closely Related Language Identification -- An Experience Report](https://mlsanity.com/paper/2602.13139/) (2026)
- [Code-Switched Language Identification is Harder Than You Think](https://mlsanity.com/paper/2402.01505/) (2024)
- [Language Fusion for Parameter-Efficient Cross-lingual Transfer](https://mlsanity.com/paper/2501.06892/) (2025)
- [DIVERS-Bench: Evaluating Language Identification Across Domain Shifts and Code-Switching](https://mlsanity.com/paper/2509.17768/) (2025)

## Abstract (arXiv)

Language identification (LID) is a fundamental step in many natural language processing pipelines. However, current LID systems are far from perfect, particularly on lower-resource languages. We present a LID model which achieves a macro-average F1 score of 0.93 and a false positive rate of 0.033 across 201 languages, outperforming previous work. We achieve this by training on a curated dataset of monolingual data, the reliability of which we ensure by auditing a sample from each source and each language manually. We make both the model and the dataset available to the research community. Finally, we carry out detailed analysis into our model's performance, both in comparison to existing open models and by language class.