How Should We Model the Probability of a Language?
Not on the map. Why?
Abstract (arXiv)
Of the over 7,000 languages spoken in the world, commercial language identification (LID) systems only reliably identify a few hundred in written form. Research-grade systems extend this coverage under certain circumstances, but for most languages coverage remains patchy or nonexistent. This position paper argues that this situation is largely self-imposed. In particular, it arises from a persistent framing of LID as decontextualized text classification, which obscures the central role of prior probability estimation and is reinforced by institutional incentives that favor global, fixed-prior models. We argue that improving coverage for tail languages requires rethinking LID as a routing problem and developing principled ways to incorporate environmental cues that make languages locally plausible.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 38 references that are in the map. First posted on arXiv 9 Feb 2026.
Builds on
14 of 38 references.
References · 14
- 2016Bag of Tricks for Efficient Text Classificationcited by 876
- 2018Learning Word Vectors for 157 Languagescited by 280
- 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
- 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 143
- 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 77
- 2023An Open Dataset and Model for Language Identificationcited by 50
- 2022AfroLID: A Neural Language Identification Tool for African Languagescited by 39
- 2021Using Radio Archives for Low-Resource Speech Recognition: Towards an Intelligent Virtual Assistant for Illiterate Userscited by 16
- 2023Findings of the VarDial Evaluation Campaign 2023cited by 14
- 2023Bhasha-Abhijnaanam: Native-script and romanized Language Identification for 22 Indic languagescited by 13
- 2023Forgotten Knowledge: Examining the Citational Amnesia in NLPcited by 13
- 2023Arabic Dialect Identification under Scrutiny: Limitations of Single-label Classificationcited by 13
Built on by
Not cited yet.