mlsanityData to 14 Oct 2026

MoVoC: Morphology-Aware Subword Construction for Geez Script Languages

arXiv 2509.08812 ↗First posted 10 Sep 2025cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Subword-based tokenization methods often fail to preserve morphological boundaries, a limitation especially pronounced in low-resource, morphologically complex languages such as those written in the Geez script. To address this, we present MoVoC (Morpheme-aware Subword Vocabulary Construction) and train MoVoC-Tok, a tokenizer that integrates supervised morphological analysis into the subword vocabulary. This hybrid segmentation approach combines morpheme-based and Byte Pair Encoding (BPE) tokens to preserve morphological integrity while maintaining lexical meaning. To tackle resource scarcity, we curate and release manually annotated morpheme data for four Geez script languages and a morpheme-aware vocabulary for two of them. While the proposed tokenization method does not lead to significant gains in automatic translation quality, we observe consistent improvements in intrinsic metrics, MorphoScore, and Boundary Precision, highlighting the value of morphology-aware segmentation in enhancing linguistic fidelity and token efficiency. Our morpheme-annotated datasets and tokenizer will be publicly available to support further research in low-resource, morphologically rich languages. Our code and data are available on GitHub: https://github.com/hailaykidu/MoVoC

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 61 references that are in the map. First posted on arXiv 10 Sep 2025.

Builds on

26 of 61 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  4. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  5. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  6. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  7. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 192
  8. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  9. 2019Neural Machine Translation with Byte-Level Subwordscited by 115
  10. 2021Survey of Low-Resource Machine Translationcited by 92
  11. 2024Tokenization Is More Than Compressioncited by 89
  12. 2023Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languagescited by 62
14 more
  1. 2024Why do language models perform worse for morphologically complex languages?cited by 62
  2. 2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61
  3. 2018Are All Languages Equally Hard to Language-Model?cited by 53
  4. 2020Morphology Matters: A Multilingual Language Modeling Analysiscited by 37
  5. 2015Multilingual Language Processing From Bytescited by 26
  6. 2025MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologiescited by 25
  7. 2018Marian: Cost-effective High-Quality Neural Machine Translation in C++cited by 22
  8. 2024Multi-word Tokenization for Sequence Compressioncited by 17
  9. 2021The Effectiveness of Morphology-aware Segmentation in Low-Resource Neural Machine Translationcited by 16
  10. 2024Lexically Grounded Subword Segmentationcited by 13
  11. 2023Effects of sub-word segmentation on performance of transformer language modelscited by 12
  12. 2024When Every Token Counts: Optimal Segmentation for Low-Resource Language Modelscited by 8
  13. 2023CompoundPiece: Evaluating and Improving Decompounding Performance of Language Modelscited by 7
  14. 2023Explicit Morphological Knowledge Improves Pre-training of Language Models for Hebrewcited by 5

Built on by

Not cited yet.