# MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies

arXiv: https://arxiv.org/abs/2502.00894 · HTML: https://mlsanity.com/paper/2502.00894/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 42 references that are in the map.
- First posted on arXiv 2 Feb 2025.

## Builds on (most cited first, 19 of 19)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [BPE-Dropout: Simple and Effective Subword Regularization](https://mlsanity.com/paper/1910.13267/) (2019)
- [Character-Level Language Modeling with Deeper Self-Attention](https://mlsanity.com/paper/1808.04444/) (2018)
- [Neural Machine Translation with Byte-Level Subwords](https://mlsanity.com/paper/1909.03341/) (2019)
- [BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting](https://mlsanity.com/paper/2212.09535/) (2022)
- [XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models](https://mlsanity.com/paper/2301.10472/) (2023)
- [Fanar: An Arabic-Centric Multimodal Generative AI Platform](https://mlsanity.com/paper/2501.13944/) (2025)
- [UniMorph 2.0: Universal Morphology](https://mlsanity.com/paper/1810.11101/) (2018)
- [Morphology Matters: A Multilingual Language Modeling Analysis](https://mlsanity.com/paper/2012.06262/) (2020)
- [Multilingual Language Processing From Bytes](https://mlsanity.com/paper/1512.00103/) (2015)
- [The SIGMORPHON 2022 Shared Task on Morpheme Segmentation](https://mlsanity.com/paper/2206.07615/) (2022)
- [Multi-word Tokenization for Sequence Compression](https://mlsanity.com/paper/2402.09949/) (2024)
- [Challenging Language-Dependent Segmentation for Arabic: An Application to Machine Translation and Part-of-Speech Tagging](https://mlsanity.com/paper/1709.00616/) (2017)
- [Subword Sampling for Low Resource Word Alignment](https://mlsanity.com/paper/2012.11657/) (2020)

## Built on by (most cited first, 25 of 25)

- [Evaluating Morphological Alignment of Tokenizers in 70 Languages](https://mlsanity.com/paper/2507.06378/) (2025)
- [The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?](https://mlsanity.com/paper/2601.07220/) (2026)
- [Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment](https://mlsanity.com/paper/2508.08424/) (2025)
- [Tokens with Meaning: A Hybrid Tokenization Approach for Turkish](https://mlsanity.com/paper/2508.14292/) (2025)
- [Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models](https://mlsanity.com/paper/2601.13260/) (2026)
- [MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment](https://mlsanity.com/paper/2606.27019/) (2026)
- [UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?](https://mlsanity.com/paper/2508.11260/) (2025)
- [Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluation](https://mlsanity.com/paper/2510.09947/) (2025)
- [Dictionaries to the Rescue: Cross-Lingual Vocabulary Transfer for Low-Resource Languages Using Bilingual Dictionaries](https://mlsanity.com/paper/2506.01535/) (2025)
- [Modelling the Morphology of Verbal Paradigms: A Case Study in the Tokenization of Turkish and Hebrew](https://mlsanity.com/paper/2602.05648/) (2026)
- [Arabic Multimodal Machine Learning: Datasets, Applications, Approaches, and Challenges](https://mlsanity.com/paper/2508.12227/) (2025)
- [Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmetic](https://mlsanity.com/paper/2510.17001/) (2025)
- [Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMs](https://mlsanity.com/paper/2603.15773/) (2026)
- [HeceTokenizer: A Syllable-Based Tokenization Approach for Turkish Retrieval](https://mlsanity.com/paper/2604.10665/) (2026)
- [MoVoC: Morphology-Aware Subword Construction for Geez Script Languages](https://mlsanity.com/paper/2509.08812/) (2025)
- [A.X K1 Technical Report](https://mlsanity.com/paper/2601.09200/) (2026)
- [Tokenization and Morphological Fidelity in Uralic NLP: A Cross-Lingual Evaluation](https://mlsanity.com/paper/2602.04241/) (2026)
- [LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation](https://mlsanity.com/paper/2603.22629/) (2026)
- [Breaking Safety at the Token Boundary: How BPE Tokenization Creates Exploitable Gaps in LLM Alignment](https://mlsanity.com/paper/2607.01239/) (2026)
- [Rate-Utility Frontiers for Language Encodings: Comparing Tokens, Bytes, and Pixels Under Controlled Linguistic Content](https://mlsanity.com/paper/2607.16117/) (2026)
- [SuTRA : Structurally-Unified Tokenization with Root Awareness](https://mlsanity.com/paper/2608.18087/) (2026)
- [The Functionalizer: Lossless Functional Decomposition for Subword Tokenization](https://mlsanity.com/paper/2609.15991/) (2026)
- [The Limits of BPE Tokenization in Polish: Segmentation-Flexional Forms, Grammatical Anchoring, and First-Person Stability in Inflectional Language Models](https://mlsanity.com/paper/2609.17553/) (2026)
- [More Than Words: Compositional Tokenization for Efficient Language Models](https://mlsanity.com/paper/2610.05597/) (2026)
- [Tracing Morphology, Semantics, and Divergence in Language Acquisition](https://mlsanity.com/paper/2610.19041/) (2026)

## Abstract (arXiv)

Tokenization is fundamental to Natural Language Processing (NLP), directly impacting model efficiency and linguistic fidelity. While Byte Pair Encoding (BPE) is widely used in Large Language Models (LLMs), it often disregards morpheme boundaries, leading to suboptimal segmentation, particularly in morphologically rich languages. We introduce MorphBPE, a morphology-aware extension of BPE that integrates linguistic structure into subword tokenization while preserving statistical efficiency. Additionally, we propose two morphology-based evaluation metrics: (i) Morphological Consistency F1-Score, which quantifies the consistency between morpheme sharing and token sharing, contributing to LLM training convergence, and (ii) Morphological Edit Distance, which measures alignment between morphemes and tokens concerning interpretability. Experiments on English, Russian, Hungarian, and Arabic across 300M and 1B parameter LLMs demonstrate that MorphBPE consistently reduces cross-entropy loss, accelerates convergence, and improves morphological alignment scores. Fully compatible with existing LLM pipelines, MorphBPE requires minimal modifications for integration. The MorphBPE codebase and tokenizer playground will be available at: https://github.com/llm-lab-org/MorphBPE and https://tokenizer.llm-lab.org