# Subword-Based Comparative Linguistics across 242 Languages Using Wikipedia Glottosets

arXiv: https://arxiv.org/abs/2601.18791 · HTML: https://mlsanity.com/paper/2601.18791/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 47 references that are in the map.
- First posted on arXiv 26 Jan 2026.

## Builds on (most cited first, 20 of 20)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [Byte Pair Encoding is Suboptimal for Language Model Pretraining](https://mlsanity.com/paper/2004.03720/) (2020)
- [Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages](https://mlsanity.com/paper/2305.12182/) (2023)
- [Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing](https://mlsanity.com/paper/1807.00914/) (2018)
- [Why do language models perform worse for morphologically complex languages?](https://mlsanity.com/paper/2411.14198/) (2024)
- [Global-scale phylogenetic linguistic inference from lexical resources](https://mlsanity.com/paper/1802.06079/) (2018)
- [Are Automatic Methods for Cognate Detection Good Enough for Phylogenetic Reconstruction in Historical Linguistics?](https://mlsanity.com/paper/1804.05416/) (2018)
- [Endangered Languages are not Low-Resourced!](https://mlsanity.com/paper/2103.09567/) (2021)
- [The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?](https://mlsanity.com/paper/2601.07220/) (2026)
- [Language Embeddings for Typology and Cross-lingual Transfer Learning](https://mlsanity.com/paper/2106.02082/) (2021)
- [Automated Cognate Detection as a Supervised Link Prediction Task with Cognate Transformer](https://mlsanity.com/paper/2402.02926/) (2024)
- [Combining SMT and NMT Back-Translated Data for Efficient NMT](https://mlsanity.com/paper/1909.03750/) (2019)
- [From Plenipotentiary to Puddingless: Users and Uses of New Words in Early English Letters](https://mlsanity.com/paper/2103.09926/) (2021)
- [Improved Cross-Lingual Transfer Learning For Automatic Speech Translation](https://mlsanity.com/paper/2306.00789/) (2023)
- [SampoNLP: A Self-Referential Toolkit for Morphological Analysis of Subword Tokenizers](https://mlsanity.com/paper/2601.04469/) (2026)
- [Processing M.A. Castr'en's Materials: Multilingual Typed and Handwritten Manuscripts](https://mlsanity.com/paper/2112.14153/) (2021)

## Abstract (arXiv)

We present a large-scale comparative study of 242 Latin and Cyrillic-script languages using subword-based methodologies. By constructing 'glottosets' from Wikipedia lexicons, we introduce a framework for simultaneous cross-linguistic comparison via Byte-Pair Encoding (BPE). Our approach utilizes rank-based subword vectors to analyze vocabulary overlap, lexical divergence, and language similarity at scale. Evaluations demonstrate that BPE segmentation aligns with morpheme boundaries 95% better than random baseline across 15 languages (F1 = 0.34 vs 0.15). BPE vocabulary similarity correlates significantly with genetic language relatedness (Mantel r = 0.329, p < 0.001), with Romance languages forming the tightest cluster (mean distance 0.51) and cross-family pairs showing clear separation (0.82). Analysis of 26,939 cross-linguistic homographs reveals that 48.7% receive different segmentations across related languages, with variation correlating to phylogenetic distance. Our results provide quantitative macro-linguistic insights into lexical patterns across typologically diverse languages within a unified analytical framework.