mlsanityData to 7 Oct 2026

Mapping Languages: The Corpus of Global Language Use

arXiv 2004.00798 ↗First posted 2 Apr 2020cs.CLcited by 42

Not on the map. Why?

Abstract (arXiv)

This paper describes a web-based corpus of global language use with a focus on how this corpus can be used for data-driven language mapping. First, the corpus provides a representation of where national varieties of major languages are used (e.g., English, Arabic, Russian) together with consistently collected data for each variety. Second, the paper evaluates a language identification model that supports more local languages with smaller sample sizes than alternative off-the-shelf models. Improved language identification is essential for moving beyond majority languages. Given the focus on language mapping, the paper analyzes how well this digital language data represents actual populations by (i) systematically comparing the corpus with demographic ground-truth data and (ii) triangulating the corpus with an alternate Twitter-based dataset. In total, the corpus contains 423 billion words representing 148 languages (with over 1 million words from each language) and 158 countries (again with over 1 million words from each country), all distilled from Common Crawl web data. The main contribution of this paper, in addition to describing this publicly-available corpus, is to provide a comprehensive analysis of the relationship between two sources of digital data (the web and Twitter) as well as their connection to underlying populations.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 8; 2023: 9; 2024: 12; 2025: 3; 2026: 5.

42 papers.

Sort:

  1. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  2. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  3. 2023An Open Dataset and Model for Language Identificationcited by 49
  4. 2022AfroLID: A Neural Language Identification Tool for African Languagescited by 38
  5. 2021Disembodied Machine Learning: On the Illusion of Objectivity in NLPcited by 37
  6. 2023GlotLID: Language Identification for Low-Resource Languagescited by 37
  7. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 27
  8. 2024EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Modelscited by 16
  9. 2024GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languagescited by 15
  10. 2021Representations of Language Varieties Are Reliable Given Corpus Similarity Measurescited by 10
  11. 2024MaskLID: Code-Switching Language Identification through Iterative Maskingcited by 10
  12. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 10
30 more
  1. 2021Learned Construction Grammars Converge Across Registers Given Increased Exposurecited by 6
  2. 2022Exposure and Emergence in Usage-Based Grammar: Computational Experiments in 35 Languagescited by 6
  3. 2024Geographically-Informed Language Identificationcited by 6
  4. 2024Pre-Trained Language Models Represent Some Geographic Populations Better Than Otherscited by 6
  5. 2022Stability of Syntactic Dialect Classification Over Space and Timecited by 5
  6. 2024Large corpora and large language models: a replicable method for automating grammatical annotationcited by 5
  7. 2025ConLID: Supervised Contrastive Learning for Low-Resource Language Identificationcited by 5
  8. 2026CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Datacited by 5
  9. 2022Corpus Similarity Measures Remain Robust Across Diverse Languagescited by 4
  10. 2023AfroDigits: A Community-Driven Spoken Digit Dataset for African Languagescited by 4
  11. 2021Capturing the diversity of multilingual societiescited by 3
  12. 2022Language Identification for Austronesian Languagescited by 3
  13. 2022Predicting Embedding Reliability in Low-Resource Settings Using Corpus Similarity Measurescited by 3
  14. 2022Register Variation Remains Stable Across 60 Languagescited by 3
  15. 2023Exploring the Constructicon: Linguistic Analysis of a Computational CxGcited by 3
  16. 2023Variation and Instability in Dialect-Based Embedding Spacescited by 3
  17. 2023cantnlp@LT-EDI-2023: Homophobia/Transphobia Detection in Social Media Comments using Spatio-Temporally Retrained Language Modelscited by 3
  18. 2024Code-Switched Language Identification is Harder Than You Thinkcited by 3
  19. 2023Comparing Measures of Linguistic Diversity Across Social Media Language Data and Census Data at Subnational Geographic Areascited by 1
  20. 2023Syntactic Variation Across the Grammar: Modelling a Complex Adaptive Systemcited by 1
  21. 2024Ethics Whitepaper: Whitepaper on Ethical Research into Large Language Modelscited by 1
  22. 2021Measuring Linguistic Diversity During COVID-19cited by 0
  23. 2024cantnlp@LT-EDI-2024: Automatic Detection of Anti-LGBTQ+ Hate Speech in Under-resourced Languagescited by 0
  24. 2024Validating and Exploring Large Geographic Corporacited by 0
  25. 2024Sociocultural Considerations in Monitoring Anti-LGBTQ+ Content on Social Mediacited by 0
  26. 2025Detecting Linguistic Diversity on Social Mediacited by 0
  27. 2025Stronger Together: Unleashing the Social Impact of Hate Speech Researchcited by 0
  28. 2026cantnlp@DravidianLangTech 2026: organic domain adaptation improves multi-class hope speech detection in Tulucited by 0
  29. 2026Improving Language Identification for Code-Switched Utterances with Integer Linear Programmingcited by 0
  30. 2026FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditingcited by 0