# Representations of Language Varieties Are Reliable Given Corpus Similarity Measures

arXiv: https://arxiv.org/abs/2104.01294 · HTML: https://mlsanity.com/paper/2104.01294/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 28 references that are in the map.
- First posted on arXiv 3 Apr 2021.

## Builds on (most cited first, 6 of 6)

- [Mapping Languages: The Corpus of Global Language Use](https://mlsanity.com/paper/2004.00798/) (2020)
- [Dialectometric analysis of language variation in Twitter](https://mlsanity.com/paper/1702.06777/) (2017)
- [Modeling Global Syntactic Variation in English Using Dialect Classification](https://mlsanity.com/paper/1904.05527/) (2019)
- [Where in the World are You? Geolocation and Language Identification in Twitter](https://mlsanity.com/paper/1308.0683/) (2013)
- [Mapping Languages and Demographics with Georeferenced Corpora](https://mlsanity.com/paper/2004.00809/) (2020)
- [Using Robust PCA to estimate regional characteristics of language use from geo-tagged Twitter messages](https://mlsanity.com/paper/1311.1169/) (2013)

## Built on by (most cited first, 10 of 10)

- [Quantifying the Dialect Gap and its Correlates Across Languages](https://mlsanity.com/paper/2310.15135/) (2023)
- [Split-and-Denoise: Protect large language model inference with local differential privacy](https://mlsanity.com/paper/2310.09130/) (2023)
- [Learned Construction Grammars Converge Across Registers Given Increased Exposure](https://mlsanity.com/paper/2110.05663/) (2021)
- [Exposure and Emergence in Usage-Based Grammar: Computational Experiments in 35 Languages](https://mlsanity.com/paper/2211.14160/) (2022)
- [Pre-Trained Language Models Represent Some Geographic Populations Better Than Others](https://mlsanity.com/paper/2403.11025/) (2024)
- [Corpus Similarity Measures Remain Robust Across Diverse Languages](https://mlsanity.com/paper/2206.04332/) (2022)
- [Predicting Embedding Reliability in Low-Resource Settings Using Corpus Similarity Measures](https://mlsanity.com/paper/2206.04330/) (2022)
- [Register Variation Remains Stable Across 60 Languages](https://mlsanity.com/paper/2209.09813/) (2022)
- [Variation and Instability in Dialect-Based Embedding Spaces](https://mlsanity.com/paper/2303.14963/) (2023)
- [Validating and Exploring Large Geographic Corpora](https://mlsanity.com/paper/2403.08198/) (2024)

## Abstract (arXiv)

This paper measures similarity both within and between 84 language varieties across nine languages. These corpora are drawn from digital sources (the web and tweets), allowing us to evaluate whether such geo-referenced corpora are reliable for modelling linguistic variation. The basic idea is that, if each source adequately represents a single underlying language variety, then the similarity between these sources should be stable across all languages and countries. The paper shows that there is a consistent agreement between these sources using frequency-based corpus similarity measures. This provides further evidence that digital geo-referenced corpora consistently represent local language varieties.