# Corpus Similarity Measures Remain Robust Across Diverse Languages

arXiv: https://arxiv.org/abs/2206.04332 · HTML: https://mlsanity.com/paper/2206.04332/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 30 references that are in the map.
- First posted on arXiv 9 Jun 2022.

## Builds on (most cited first, 5 of 5)

- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [Mapping Languages: The Corpus of Global Language Use](https://mlsanity.com/paper/2004.00798/) (2020)
- [Global Syntactic Variation in Seven Languages: Towards a Computational Dialectology](https://mlsanity.com/paper/2104.01306/) (2021)
- [Representations of Language Varieties Are Reliable Given Corpus Similarity Measures](https://mlsanity.com/paper/2104.01294/) (2021)
- [A German Corpus for Text Similarity Detection Tasks](https://mlsanity.com/paper/1703.03923/) (2017)

## Built on by (most cited first, 4 of 4)

- [Pre-Trained Language Models Represent Some Geographic Populations Better Than Others](https://mlsanity.com/paper/2403.11025/) (2024)
- [Register Variation Remains Stable Across 60 Languages](https://mlsanity.com/paper/2209.09813/) (2022)
- [UNIDECOR: A Unified Deception Corpus for Cross-Corpus Deception Detection](https://mlsanity.com/paper/2306.02827/) (2023)
- [Validating and Exploring Large Geographic Corpora](https://mlsanity.com/paper/2403.08198/) (2024)

## Abstract (arXiv)

This paper experiments with frequency-based corpus similarity measures across 39 languages using a register prediction task. The goal is to quantify (i) the distance between different corpora from the same language and (ii) the homogeneity of individual corpora. Both of these goals are essential for measuring how well corpus-based linguistic analysis generalizes from one dataset to another. The problem is that previous work has focused on Indo-European languages, raising the question of whether these measures are able to provide robust generalizations across diverse languages. This paper uses a register prediction task to evaluate competing measures across 39 languages: how well are they able to distinguish between corpora representing different contexts of production? Each experiment compares three corpora from a single language, with the same three digital registers shared across all languages: social media, web pages, and Wikipedia. Results show that measures of corpus similarity retain their validity across different language families, writing systems, and types of morphology. Further, the measures remain robust when evaluated on out-of-domain corpora, when applied to low-resource languages, and when applied to different sets of registers. These findings are significant given our need to make generalizations across the rapidly increasing number of corpora available for analysis.