# Mapping Languages: The Corpus of Global Language Use

arXiv: https://arxiv.org/abs/2004.00798 · HTML: https://mlsanity.com/paper/2004.00798/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2020.

## Built on by (most cited first, 30 of 42)

- [Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages](https://mlsanity.com/paper/2305.12182/) (2023)
- [Building Machine Translation Systems for the Next Thousand Languages](https://mlsanity.com/paper/2205.03983/) (2022)
- [An Open Dataset and Model for Language Identification](https://mlsanity.com/paper/2305.13820/) (2023)
- [AfroLID: A Neural Language Identification Tool for African Languages](https://mlsanity.com/paper/2210.11744/) (2022)
- [GlotLID: Language Identification for Low-Resource Languages](https://mlsanity.com/paper/2310.16248/) (2023)
- [Disembodied Machine Learning: On the Illusion of Objectivity in NLP](https://mlsanity.com/paper/2101.11974/) (2021)
- [Goldfish: Monolingual Language Models for 350 Languages](https://mlsanity.com/paper/2408.10441/) (2024)
- [EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models](https://mlsanity.com/paper/2409.17892/) (2024)
- [GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages](https://mlsanity.com/paper/2410.23825/) (2024)
- [MaskLID: Code-Switching Language Identification through Iterative Masking](https://mlsanity.com/paper/2406.06263/) (2024)
- [The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?](https://mlsanity.com/paper/2601.07220/) (2026)
- [Representations of Language Varieties Are Reliable Given Corpus Similarity Measures](https://mlsanity.com/paper/2104.01294/) (2021)
- [Learned Construction Grammars Converge Across Registers Given Increased Exposure](https://mlsanity.com/paper/2110.05663/) (2021)
- [Exposure and Emergence in Usage-Based Grammar: Computational Experiments in 35 Languages](https://mlsanity.com/paper/2211.14160/) (2022)
- [Geographically-Informed Language Identification](https://mlsanity.com/paper/2403.09892/) (2024)
- [Pre-Trained Language Models Represent Some Geographic Populations Better Than Others](https://mlsanity.com/paper/2403.11025/) (2024)
- [ConLID: Supervised Contrastive Learning for Low-Resource Language Identification](https://mlsanity.com/paper/2506.15304/) (2025)
- [CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data](https://mlsanity.com/paper/2601.18026/) (2026)
- [Stability of Syntactic Dialect Classification Over Space and Time](https://mlsanity.com/paper/2209.04958/) (2022)
- [Large corpora and large language models: a replicable method for automating grammatical annotation](https://mlsanity.com/paper/2411.11260/) (2024)
- [Corpus Similarity Measures Remain Robust Across Diverse Languages](https://mlsanity.com/paper/2206.04332/) (2022)
- [AfroDigits: A Community-Driven Spoken Digit Dataset for African Languages](https://mlsanity.com/paper/2303.12582/) (2023)
- [Capturing the diversity of multilingual societies](https://mlsanity.com/paper/2105.02570/) (2021)
- [Language Identification for Austronesian Languages](https://mlsanity.com/paper/2206.04327/) (2022)
- [Predicting Embedding Reliability in Low-Resource Settings Using Corpus Similarity Measures](https://mlsanity.com/paper/2206.04330/) (2022)
- [Register Variation Remains Stable Across 60 Languages](https://mlsanity.com/paper/2209.09813/) (2022)
- [Exploring the Constructicon: Linguistic Analysis of a Computational CxG](https://mlsanity.com/paper/2301.12642/) (2023)
- [Variation and Instability in Dialect-Based Embedding Spaces](https://mlsanity.com/paper/2303.14963/) (2023)
- [cantnlp@LT-EDI-2023: Homophobia/Transphobia Detection in Social Media Comments using Spatio-Temporally Retrained Language Models](https://mlsanity.com/paper/2308.10370/) (2023)
- [Code-Switched Language Identification is Harder Than You Think](https://mlsanity.com/paper/2402.01505/) (2024)

## Abstract (arXiv)

This paper describes a web-based corpus of global language use with a focus on how this corpus can be used for data-driven language mapping. First, the corpus provides a representation of where national varieties of major languages are used (e.g., English, Arabic, Russian) together with consistently collected data for each variety. Second, the paper evaluates a language identification model that supports more local languages with smaller sample sizes than alternative off-the-shelf models. Improved language identification is essential for moving beyond majority languages. Given the focus on language mapping, the paper analyzes how well this digital language data represents actual populations by (i) systematically comparing the corpus with demographic ground-truth data and (ii) triangulating the corpus with an alternate Twitter-based dataset. In total, the corpus contains 423 billion words representing 148 languages (with over 1 million words from each language) and 158 countries (again with over 1 million words from each country), all distilled from Common Crawl web data. The main contribution of this paper, in addition to describing this publicly-available corpus, is to provide a comprehensive analysis of the relationship between two sources of digital data (the web and Twitter) as well as their connection to underlying populations.