# Beyond Literal Token Overlap: Token Alignability for Multilinguality

arXiv: https://arxiv.org/abs/2502.06468 · HTML: https://mlsanity.com/paper/2502.06468/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Builds on (most cited first, 29 of 29)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs](https://mlsanity.com/paper/1911.06154/) (2019)
- [Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialog](https://mlsanity.com/paper/1810.13327/) (2018)
- [Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks](https://mlsanity.com/paper/1909.00964/) (2019)
- [On the Language Neutrality of Pre-trained Multilingual Representations](https://mlsanity.com/paper/2004.05160/) (2020)
- [First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT](https://mlsanity.com/paper/2101.11109/) (2021)
- [Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages](https://mlsanity.com/paper/2305.17179/) (2023)
- [When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer](https://mlsanity.com/paper/2110.14782/) (2021)
- [MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling](https://mlsanity.com/paper/2403.10691/) (2024)
- [Understanding Cross-Lingual Alignment -- A Survey](https://mlsanity.com/paper/2404.06228/) (2024)
- [MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization](https://mlsanity.com/paper/2407.08818/) (2024)
- [mOthello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models?](https://mlsanity.com/paper/2404.12444/) (2024)
- [A Vocabulary-Free Multilingual Neural Tokenizer for End-to-End Task Learning](https://mlsanity.com/paper/2204.10815/) (2022)
- [The Role of Language Imbalance in Cross-lingual Generalisation: Insights from Cloned Language Experiments](https://mlsanity.com/paper/2404.07982/) (2024)
- [A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space](https://mlsanity.com/paper/2109.06324/) (2021)
- [Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge](https://mlsanity.com/paper/2404.13292/) (2024)

## Built on by (most cited first, 9 of 9)

- [False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models](https://mlsanity.com/paper/2509.18750/) (2025)
- [Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insights](https://mlsanity.com/paper/2506.17789/) (2025)
- [Uncovering inequalities in new knowledge learning by large language models across different languages](https://mlsanity.com/paper/2503.04064/) (2025)
- [Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features](https://mlsanity.com/paper/2601.18536/) (2026)
- [Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining](https://mlsanity.com/paper/2604.17633/) (2026)
- [Conditional Unigram Tokenization with Parallel Data](https://mlsanity.com/paper/2507.07824/) (2025)
- [Toward Culturally Grounded Natural Language Processing](https://mlsanity.com/paper/2603.26013/) (2026)
- [An In-Vitro Study on Cross-Lingual Generalization in Language Models](https://mlsanity.com/paper/2605.26683/) (2026)
- [Predicting Multilingual Classification and Translation Performance of LLMs with Cross-Lingual Alignment -- Is English Enough?](https://mlsanity.com/paper/2608.03446/) (2026)

## Abstract (arXiv)

Previous work has considered token overlap, or even similarity of token distributions, as predictors for multilinguality and cross-lingual knowledge transfer in language models. However, these very literal metrics assign large distances to language pairs with different scripts, which can nevertheless show good cross-linguality. This limits the explanatory strength of token overlap for knowledge transfer between language pairs that use distinct scripts or follow different orthographic conventions. In this paper, we propose subword token alignability as a new way to understand the impact and quality of multilingual tokenisation. In particular, this metric predicts multilinguality much better when scripts are disparate and the overlap of literal tokens is low. We analyse this metric in the context of both encoder and decoder models, look at data size as a potential distractor, and discuss how this insight may be applied to multilingual tokenisation in future work. We recommend our subword token alignability metric for identifying optimal language pairs for cross-lingual transfer, as well as to guide the construction of better multilingual tokenisers in the future. We publish our code and reproducibility details.