mlsanityData to 14 Oct 2026

Subword-Based Comparative Linguistics across 242 Languages Using Wikipedia Glottosets

arXiv 2601.18791 ↗First posted 26 Jan 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We present a large-scale comparative study of 242 Latin and Cyrillic-script languages using subword-based methodologies. By constructing 'glottosets' from Wikipedia lexicons, we introduce a framework for simultaneous cross-linguistic comparison via Byte-Pair Encoding (BPE). Our approach utilizes rank-based subword vectors to analyze vocabulary overlap, lexical divergence, and language similarity at scale. Evaluations demonstrate that BPE segmentation aligns with morpheme boundaries 95% better than random baseline across 15 languages (F1 = 0.34 vs 0.15). BPE vocabulary similarity correlates significantly with genetic language relatedness (Mantel r = 0.329, p < 0.001), with Romance languages forming the tightest cluster (mean distance 0.51) and cross-family pairs showing clear separation (0.82). Analysis of 26,939 cross-linguistic homographs reveals that 48.7% receive different segmentations across related languages, with variation correlating to phylogenetic distance. Our results provide quantitative macro-linguistic insights into lexical patterns across typologically diverse languages within a unified analytical framework.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 47 references that are in the map. First posted on arXiv 26 Jan 2026.

Builds on

20 of 47 references.

References · 20

  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  3. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  4. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
  5. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  6. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 192
  7. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  8. 2018Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processingcited by 63
  9. 2024Why do language models perform worse for morphologically complex languages?cited by 62
  10. 2018Global-scale phylogenetic linguistic inference from lexical resourcescited by 15
  11. 2018Are Automatic Methods for Cognate Detection Good Enough for Phylogenetic Reconstruction in Historical Linguistics?cited by 15
  12. 2021Endangered Languages are not Low-Resourced!cited by 14
8 more
  1. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 11
  2. 2021Language Embeddings for Typology and Cross-lingual Transfer Learningcited by 8
  3. 2024Automated Cognate Detection as a Supervised Link Prediction Task with Cognate Transformercited by 6
  4. 2019Combining SMT and NMT Back-Translated Data for Efficient NMTcited by 4
  5. 2021From Plenipotentiary to Puddingless: Users and Uses of New Words in Early English Letterscited by 4
  6. 2023Improved Cross-Lingual Transfer Learning For Automatic Speech Translationcited by 2
  7. 2026SampoNLP: A Self-Referential Toolkit for Morphological Analysis of Subword Tokenizerscited by 2
  8. 2021Processing M.A. Castr'en's Materials: Multilingual Typed and Handwritten Manuscriptscited by 1

Built on by

Not cited yet.