# The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languages

arXiv: https://arxiv.org/abs/2511.09197 · HTML: https://mlsanity.com/paper/2511.09197/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 43 references that are in the map.
- First posted on arXiv 12 Nov 2025.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [BPE-Dropout: Simple and Effective Subword Regularization](https://mlsanity.com/paper/1910.13267/) (2019)
- [A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation](https://mlsanity.com/paper/2205.02022/) (2022)
- [Probing Across Time: What Does RoBERTa Know and When?](https://mlsanity.com/paper/2104.07885/) (2021)
- [The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation](https://mlsanity.com/paper/2110.03036/) (2021)
- [Language acquisition: do children and language models follow similar learning stages?](https://mlsanity.com/paper/2306.03586/) (2023)
- [Multi-view Subword Regularization](https://mlsanity.com/paper/2103.08490/) (2021)
- [Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translation](https://mlsanity.com/paper/2005.06606/) (2020)
- [Pretrained Language Model Embryology: The Birth of ALBERT](https://mlsanity.com/paper/2010.02480/) (2020)
- [The SIGMORPHON 2022 Shared Task on Morpheme Segmentation](https://mlsanity.com/paper/2206.07615/) (2022)
- [Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training](https://mlsanity.com/paper/2310.16484/) (2023)
- [Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation](https://mlsanity.com/paper/2302.14220/) (2023)
- [Unsupervised Neural Word Segmentation for Chinese via Segmental Language Modeling](https://mlsanity.com/paper/1810.03167/) (2018)
- [Learning to Segment Inputs for NMT Favors Character-Level Processing](https://mlsanity.com/paper/1810.01480/) (2018)
- [On the Importance of Subword Information for Morphological Tasks in Truly Low-Resource Languages](https://mlsanity.com/paper/1909.12375/) (2019)
- [A Masked Segmental Language Model for Unsupervised Natural Language Segmentation](https://mlsanity.com/paper/2104.07829/) (2021)
- [Subword Segmental Language Modelling for Nguni Languages](https://mlsanity.com/paper/2210.06525/) (2022)
- [Subword-Delimited Downsampling for Better Character-Level Translation](https://mlsanity.com/paper/2212.01304/) (2022)
- [Subword Segmental Machine Translation: Unifying Segmentation and Target Sentence Generation](https://mlsanity.com/paper/2305.07005/) (2023)
- [PuoBERTa: Training and evaluation of a curated language model for Setswana](https://mlsanity.com/paper/2310.09141/) (2023)
- [Triples-to-isiXhosa (T2X): Addressing the Challenges of Low-Resource Agglutinative Data-to-Text Generation](https://mlsanity.com/paper/2403.07567/) (2024)

## Built on by (most cited first, 2 of 2)

- [What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling?](https://mlsanity.com/paper/2608.17325/) (2026)
- [Subword Segmental BabyLMs: Learning to Tokenise for Sample-Efficient Pretraining](https://mlsanity.com/paper/2609.01151/) (2026)

## Abstract (arXiv)

Subword segmentation is typically applied in preprocessing and stays fixed during training. Alternatively, it can be learned during training to optimise the training objective. In this paper we study the learning dynamics of subword segmentation: if a language model can dynamically optimise tokenisation, how do its subwords evolve during pretraining and finetuning? To explore this, we extend the subword segmental language model (SSLM), a framework for learning subwords during training, to support pretraining and finetuning. We train models for three typologically diverse languages to study learning dynamics across the morphological spectrum: Isi-Xhosa is conjunctive (long word forms composed of many morphemes), Setswana is disjunctive (morphemes written as separate words), and English represents a typological middle ground. We analyse subword dynamics from a linguistic perspective, tracking morphology, productivity, and fertility. We identify four stages of subword learning, with the morphologically complex isi-Xhosa exhibiting greater instability. During finetuning, subword boundaries shift to become finer-grained. Lastly, we show that learnable subwords offers a promising approach to improve text generation and cross-lingual transfer for low-resource, morphologically complex languages.