# How Should We Model the Probability of a Language?

arXiv: https://arxiv.org/abs/2602.08951 · HTML: https://mlsanity.com/paper/2602.08951/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 38 references that are in the map.
- First posted on arXiv 9 Feb 2026.

## Builds on (most cited first, 14 of 14)

- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Learning Word Vectors for 157 Languages](https://mlsanity.com/paper/1802.06893/) (2018)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language](https://mlsanity.com/paper/2506.20920/) (2025)
- [Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus](https://mlsanity.com/paper/2010.14571/) (2020)
- [An Open Dataset and Model for Language Identification](https://mlsanity.com/paper/2305.13820/) (2023)
- [AfroLID: A Neural Language Identification Tool for African Languages](https://mlsanity.com/paper/2210.11744/) (2022)
- [Using Radio Archives for Low-Resource Speech Recognition: Towards an Intelligent Virtual Assistant for Illiterate Users](https://mlsanity.com/paper/2104.13083/) (2021)
- [Findings of the VarDial Evaluation Campaign 2023](https://mlsanity.com/paper/2305.20080/) (2023)
- [Bhasha-Abhijnaanam: Native-script and romanized Language Identification for 22 Indic languages](https://mlsanity.com/paper/2305.15814/) (2023)
- [Forgotten Knowledge: Examining the Citational Amnesia in NLP](https://mlsanity.com/paper/2305.18554/) (2023)
- [Arabic Dialect Identification under Scrutiny: Limitations of Single-label Classification](https://mlsanity.com/paper/2310.13661/) (2023)
- [LIMIT: Language Identification, Misidentification, and Translation using Hierarchical Models in 350+ Languages](https://mlsanity.com/paper/2305.14263/) (2023)
- [Geographically-Informed Language Identification](https://mlsanity.com/paper/2403.09892/) (2024)

## Abstract (arXiv)

Of the over 7,000 languages spoken in the world, commercial language identification (LID) systems only reliably identify a few hundred in written form. Research-grade systems extend this coverage under certain circumstances, but for most languages coverage remains patchy or nonexistent. This position paper argues that this situation is largely self-imposed. In particular, it arises from a persistent framing of LID as decontextualized text classification, which obscures the central role of prior probability estimation and is reinforced by institutional incentives that favor global, fixed-prior models. We argue that improving coverage for tail languages requires rethinking LID as a routing problem and developing principled ways to incorporate environmental cues that make languages locally plausible.