# Better Character Language Modeling Through Morphology

arXiv: https://arxiv.org/abs/1906.01037 · HTML: https://mlsanity.com/paper/1906.01037/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jun 2019.

## Built on by (most cited first, 9 of 9)

- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [A multilabel approach to morphosyntactic probing](https://mlsanity.com/paper/2104.08464/) (2021)
- [Morphological Processing of Low-Resource Languages: Where We Are and What's Next](https://mlsanity.com/paper/2203.08909/) (2022)
- [Data-driven Model Generalizability in Crosslinguistic Low-resource Morphological Segmentation](https://mlsanity.com/paper/2201.01845/) (2022)
- [Discrete Diffusion Models for Language Generation](https://mlsanity.com/paper/2507.07050/) (2025)
- [PRISM: Distributed Inference for Foundation Models at Edge](https://mlsanity.com/paper/2507.12145/) (2025)
- [Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmetic](https://mlsanity.com/paper/2510.17001/) (2025)
- [Revisiting Syllables in Language Modelling and their Application on Low-Resource Machine Translation](https://mlsanity.com/paper/2210.02509/) (2022)
- [More Than Words: Compositional Tokenization for Efficient Language Models](https://mlsanity.com/paper/2610.05597/) (2026)

## Abstract (arXiv)

We incorporate morphological supervision into character language models (CLMs) via multitasking and show that this addition improves bits-per-character (BPC) performance across 24 languages, even when the morphology data and language modeling data are disjoint. Analyzing the CLMs shows that inflected words benefit more from explicitly modeling morphology than uninflected words, and that morphological supervision improves performance even as the amount of language modeling data grows. We then transfer morphological supervision across languages to improve language modeling performance in the low-resource setting.