# From Smor-re-brod to Subwords: Training LLMs on Danish, One Morpheme at a Time

arXiv: https://arxiv.org/abs/2504.01540 · HTML: https://mlsanity.com/paper/2504.01540/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 38 references that are in the map.
- First posted on arXiv 2 Apr 2025.

## Builds on (most cited first, 13 of 13)

- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [Byte Pair Encoding is Suboptimal for Language Model Pretraining](https://mlsanity.com/paper/2004.03720/) (2020)
- [CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages](https://mlsanity.com/paper/2309.09400/) (2023)
- [Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster](https://mlsanity.com/paper/2304.03208/) (2023)
- [ConvoKit: A Toolkit for the Analysis of Conversations](https://mlsanity.com/paper/2005.04246/) (2020)
- [Impact of Tokenization on Language Models: An Analysis for Turkish](https://mlsanity.com/paper/2204.08832/) (2022)
- [MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling](https://mlsanity.com/paper/2403.10691/) (2024)
- [Morphological and Language-Agnostic Word Segmentation for NMT](https://mlsanity.com/paper/1806.05482/) (2018)
- [Morphological Word Segmentation on Agglutinative Languages for Neural Machine Translation](https://mlsanity.com/paper/2001.01589/) (2020)
- [FT Speech: Danish Parliament Speech Corpus](https://mlsanity.com/paper/2005.12368/) (2020)
- [Introducing Syllable Tokenization for Low-resource Languages: A Case Study with Swahili](https://mlsanity.com/paper/2406.15358/) (2024)

## Abstract (arXiv)

The best performing transformer-based language models use subword tokenization techniques, such as Byte-Pair-Encoding (BPE). However, these approaches often overlook linguistic principles, such as morphological segmentation, which we believe is fundamental for understanding language-specific word structure. In this study, we leverage an annotated Danish morphological dataset to train a semisupervised model for morphological segmentation, enabling the development of tokenizers optimized for Danish morphology. We evaluate four distinct tokenizers, including two custom morphological tokenizers, by analyzing their performance in morphologically segmenting Danish words. Additionally, we train two generative transformer models, textitCerebrasGPT-111M and textitLLaMA-3.2 1B, using these tokenizers and evaluate their downstream performance. Our findings reveal that our custom-developed tokenizers substantially enhance morphological segmentation, achieving an F1 score of 58.84, compared to 39.28 achieved by a Danish BPE tokenizer. In downstream tasks, models trained with our morphological tokenizers outperform those using BPE tokenizers across different evaluation metrics. These results highlight that incorporating Danish morphological segmentation strategies into tokenizers leads to improved performance in generative transformer models on Danish language