# Subword Segmental Language Modelling for Nguni Languages

arXiv: https://arxiv.org/abs/2210.06525 · HTML: https://mlsanity.com/paper/2210.06525/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 38 references that are in the map.
- First posted on arXiv 12 Oct 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [Multi-view Subword Regularization](https://mlsanity.com/paper/2103.08490/) (2021)
- [KinyaBERT: a Morphology-aware Kinyarwanda Language Model](https://mlsanity.com/paper/2203.08459/) (2022)
- [Sequence Modeling via Segmentations](https://mlsanity.com/paper/1702.07463/) (2017)
- [Segmental Recurrent Neural Networks](https://mlsanity.com/paper/1511.06018/) (2015)
- [Optimizing Segmentation Granularity for Neural Machine Translation](https://mlsanity.com/paper/1810.08641/) (2018)
- [Unsupervised Neural Word Segmentation for Chinese via Segmental Language Modeling](https://mlsanity.com/paper/1810.03167/) (2018)
- [On the Importance of Subword Information for Morphological Tasks in Truly Low-Resource Languages](https://mlsanity.com/paper/1909.12375/) (2019)
- [Canonical and Surface Morphological Segmentation for Nguni Languages](https://mlsanity.com/paper/2104.00767/) (2021)
- [Low-Resource Language Modelling of South African Languages](https://mlsanity.com/paper/2104.00772/) (2021)
- [A Masked Segmental Language Model for Unsupervised Natural Language Segmentation](https://mlsanity.com/paper/2104.07829/) (2021)
- [A Systematic Study of Leveraging Subword Information for Learning Word Representations](https://mlsanity.com/paper/1904.07994/) (2019)
- [Unsupervised Word Segmentation with Bi-directional Neural Language Model](https://mlsanity.com/paper/2103.01421/) (2021)

## Built on by (most cited first, 8 of 8)

- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [Subword Segmental Machine Translation: Unifying Segmentation and Target Sentence Generation](https://mlsanity.com/paper/2305.07005/) (2023)
- [PuoBERTa: Training and evaluation of a curated language model for Setswana](https://mlsanity.com/paper/2310.09141/) (2023)
- [Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models](https://mlsanity.com/paper/2601.13260/) (2026)
- [Triples-to-isiXhosa (T2X): Addressing the Challenges of Low-Resource Agglutinative Data-to-Text Generation](https://mlsanity.com/paper/2403.07567/) (2024)
- [The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languages](https://mlsanity.com/paper/2511.09197/) (2025)
- [What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling?](https://mlsanity.com/paper/2608.17325/) (2026)
- [Subword Segmental BabyLMs: Learning to Tokenise for Sample-Efficient Pretraining](https://mlsanity.com/paper/2609.01151/) (2026)

## Abstract (arXiv)

Subwords have become the standard units of text in NLP, enabling efficient open-vocabulary models. With algorithms like byte-pair encoding (BPE), subword segmentation is viewed as a preprocessing step applied to the corpus before training. This can lead to sub-optimal segmentations for low-resource languages with complex morphologies. We propose a subword segmental language model (SSLM) that learns how to segment words while being trained for autoregressive language modelling. By unifying subword segmentation and language modelling, our model learns subwords that optimise LM performance. We train our model on the 4 Nguni languages of South Africa. These are low-resource agglutinative languages, so subword information is critical. As an LM, SSLM outperforms existing approaches such as BPE-based models on average across the 4 languages. Furthermore, it outperforms standard subword segmenters on unsupervised morphological segmentation. We also train our model as a word-level sequence model, resulting in an unsupervised morphological segmenter that outperforms existing methods by a large margin for all 4 languages. Our results show that learning subword segmentation is an effective alternative to existing subword segmenters, enabling the model to discover morpheme-like subwords that improve its LM capabilities.