# Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features

arXiv: https://arxiv.org/abs/2601.18536 · HTML: https://mlsanity.com/paper/2601.18536/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 26 Jan 2026.

## Builds on (most cited first, 13 of 13)

- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [Tokenization Is More Than Compression](https://mlsanity.com/paper/2402.18376/) (2024)
- [Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages](https://mlsanity.com/paper/2305.17179/) (2023)
- [Why do language models perform worse for morphologically complex languages?](https://mlsanity.com/paper/2411.14198/) (2024)
- [Tokenization and the Noiseless Channel](https://mlsanity.com/paper/2306.16842/) (2023)
- [Greed is All You Need: An Evaluation of Tokenizer Inference Methods](https://mlsanity.com/paper/2403.01289/) (2024)
- [The SIGMORPHON 2022 Shared Task on Morpheme Segmentation](https://mlsanity.com/paper/2206.07615/) (2022)
- [Lexically Grounded Subword Segmentation](https://mlsanity.com/paper/2406.13560/) (2024)
- [Beyond Literal Token Overlap: Token Alignability for Multilinguality](https://mlsanity.com/paper/2502.06468/) (2025)
- [Tokenization with Factorized Subword Encoding](https://mlsanity.com/paper/2306.07764/) (2023)
- [CUNI Systems for the WMT22 Czech-Ukrainian Translation Task](https://mlsanity.com/paper/2212.00486/) (2022)

## Built on by (most cited first, 1 of 1)

- [MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment](https://mlsanity.com/paper/2606.27019/) (2026)

## Abstract (arXiv)

We present a novel metric for the evaluation of the morphological plausibility of subword segmentation. Unlike the typically used morpheme boundary or retrieval F-score, which requires gold segmentation data that is either unavailable or of inconsistent quality across many languages, our approach utilizes morpho-syntactic features. These are available in resources such as Universal Dependencies or UniMorph for a much wider range of languages. The metric works by probabilistically aligning subwords with morphological features through an IBM Model 1. Our experiments show that the metric correlates well with traditional morpheme boundary recall while being more broadly applicable across languages with different morphological systems.