# Validating and Exploring Large Geographic Corpora

arXiv: https://arxiv.org/abs/2403.08198 · HTML: https://mlsanity.com/paper/2403.08198/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 32 references that are in the map.
- First posted on arXiv 13 Mar 2024.

## Builds on (most cited first, 11 of 11)

- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Mapping Languages: The Corpus of Global Language Use](https://mlsanity.com/paper/2004.00798/) (2020)
- [Dataset Geography: Mapping Language Data to Language Users](https://mlsanity.com/paper/2112.03497/) (2021)
- [Modeling Global Syntactic Variation in English Using Dialect Classification](https://mlsanity.com/paper/1904.05527/) (2019)
- [Finding Variants for Construction-Based Dialectometry: A Corpus-Based Approach to Regional CxGs](https://mlsanity.com/paper/2104.01299/) (2021)
- [Global Syntactic Variation in Seven Languages: Towards a Computational Dialectology](https://mlsanity.com/paper/2104.01306/) (2021)
- [Representations of Language Varieties Are Reliable Given Corpus Similarity Measures](https://mlsanity.com/paper/2104.01294/) (2021)
- [Document Classification by Inversion of Distributed Language Representations](https://mlsanity.com/paper/1504.07295/) (2015)
- [Corpus Similarity Measures Remain Robust Across Diverse Languages](https://mlsanity.com/paper/2206.04332/) (2022)
- [Register Variation Remains Stable Across 60 Languages](https://mlsanity.com/paper/2209.09813/) (2022)

## Abstract (arXiv)

This paper investigates the impact of corpus creation decisions on large multi-lingual geographic web corpora. Beginning with a 427 billion word corpus derived from the Common Crawl, three methods are used to improve the quality of sub-corpora representing specific language-country pairs like New Zealand English: (i) the agreement of independent language identification systems, (ii) hash-based deduplication, and (iii) location-specific outlier detection. The impact of each of these steps is then evaluated at the language level and the country level by using corpus similarity measures to compare each resulting corpus with baseline data sets. The goal is to understand the impact of upstream data cleaning decisions on downstream corpora with a specific focus on under-represented languages and populations. The evaluation shows that the validity of sub-corpora is improved with each stage of cleaning but that this improvement is unevenly distributed across languages and populations. This result shows how standard corpus creation techniques can accidentally exclude under-represented populations.