# CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models

arXiv: https://arxiv.org/abs/2305.14214 · HTML: https://mlsanity.com/paper/2305.14214/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 71 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [Scaling Up Models and Data with textttt5x and textttseqio](https://mlsanity.com/paper/2203.17189/) (2022)
- [Charformer: Fast Character Transformers via Gradient-based Subword Tokenization](https://mlsanity.com/paper/2106.12672/) (2021)
- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://mlsanity.com/paper/2112.10508/) (2021)
- [Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words](https://mlsanity.com/paper/2101.00403/) (2021)
- [The SIGMORPHON 2019 Shared Task: Morphological Analysis in Context and Cross-Lingual Transfer for Inflection](https://mlsanity.com/paper/1910.11493/) (2019)
- [The CoNLL--SIGMORPHON 2018 Shared Task: Universal Morphological Reinflection](https://mlsanity.com/paper/1810.07125/) (2018)
- [How Much Does Tokenization Affect Neural Machine Translation?](https://mlsanity.com/paper/1812.08621/) (2018)
- [The SIGMORPHON 2022 Shared Task on Morpheme Segmentation](https://mlsanity.com/paper/2206.07615/) (2022)
- [(Un)solving Morphological Inflection: Lemma Overlap Artificially Inflates Models' Performance](https://mlsanity.com/paper/2108.05682/) (2021)
- [The Effectiveness of Morphology-aware Segmentation in Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2103.11189/) (2021)
- [Morphological Inflection Generation Using Character Sequence to Sequence Learning](https://mlsanity.com/paper/1512.06110/) (2015)
- [Morphological Word Segmentation on Agglutinative Languages for Neural Machine Translation](https://mlsanity.com/paper/2001.01589/) (2020)
- [UniMorph 4.0: Universal Morphology](https://mlsanity.com/paper/2205.03608/) (2022)
- [DEMorphy, German Language Morphological Analyzer](https://mlsanity.com/paper/1803.00902/) (2018)

## Built on by (most cited first, 7 of 7)

- [Why do language models perform worse for morphologically complex languages?](https://mlsanity.com/paper/2411.14198/) (2024)
- [Greed is All You Need: An Evaluation of Tokenizer Inference Methods](https://mlsanity.com/paper/2403.01289/) (2024)
- [Bolmo: Byteifying the Next Generation of Language Models](https://mlsanity.com/paper/2512.15586/) (2025)
- [Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge](https://mlsanity.com/paper/2404.13292/) (2024)
- [ByteSpan: Information-Driven Subword Tokenisation](https://mlsanity.com/paper/2506.18639/) (2025)
- [Unsupervised Morphological Tree Tokenizer](https://mlsanity.com/paper/2406.15245/) (2024)
- [MoVoC: Morphology-Aware Subword Construction for Geez Script Languages](https://mlsanity.com/paper/2509.08812/) (2025)

## Abstract (arXiv)

While many languages possess processes of joining two or more words to create compound words, previous studies have been typically limited only to languages with excessively productive compound formation (e.g., German, Dutch) and there is no public dataset containing compound and non-compound words across a large number of languages. In this work, we systematically study decompounding, the task of splitting compound words into their constituents, at a wide scale. We first address the data gap by introducing a dataset of 255k compound and non-compound words across 56 diverse languages obtained from Wiktionary. We then use this dataset to evaluate an array of Large Language Models (LLMs) on the decompounding task. We find that LLMs perform poorly, especially on words which are tokenized unfavorably by subword tokenization. We thus introduce a novel methodology to train dedicated models for decompounding. The proposed two-stage procedure relies on a fully self-supervised objective in the first stage, while the second, supervised learning stage optionally fine-tunes the model on the annotated Wiktionary data. Our self-supervised models outperform the prior best unsupervised decompounding models by 13.9% accuracy on average. Our fine-tuned models outperform all prior (language-specific) decompounding tools. Furthermore, we use our models to leverage decompounding during the creation of a subword tokenizer, which we refer to as CompoundPiece. CompoundPiece tokenizes compound words more favorably on average, leading to improved performance on decompounding over an otherwise equivalent model using SentencePiece tokenization.