mlsanityData to 7 Oct 2026

MoirfEolas and Cr'iochScore: Developing Resources for and the Evaluation of Tokenization Alignment with Irish Morphology

arXiv 2609.05022 ↗First posted 4 Sep 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper presents new tokenization resources for Irish and evaluation measures of alignment with the morphological boundaries of the language. We present MoirfEolas, a dataset of over 35,000 Irish words mapped to their respective eclipses, prefixes and suffixes as well as an evaluation metric Cr'iochScore, that evaluates the alignment of tokenizations with the morphological boundaries present in MoirfEolas. We evaluate common tokenization algorithms using Cr'iochScore as well as intrinsic metrics present in the tokenization literature. We find that the Unigram Language Model aligns with Irish morphology more often than the other algorithms evaluated. We also find trade-offs between morphological-alignment of tokenization with both compression as well as vocabulary efficiency, providing practical insights for Irish natural language processing development. This dataset contributes towards combating the Irish language's low-resource status; moreover, the construction process reported in this paper can be emulated by other languages to create specialised morphological resources.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 43 references that are in the map. First posted on arXiv 4 Sep 2026.

Builds on

28 of 43 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  9. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  10. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  11. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  12. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
16 more
  1. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  2. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 105
  3. 2024Tokenization Is More Than Compressioncited by 87
  4. 2024Why do language models perform worse for morphologically complex languages?cited by 62
  5. 2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61
  6. 2024A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languagescited by 22
  7. 2025Evaluating Morphological Alignment of Tokenizers in 70 Languagescited by 22
  8. 2024Tokenization Falling Short: On Subword Robustness in Large Language Modelscited by 18
  9. 2025How Much is Enough? The Diminishing Returns of Tokenization Training Datacited by 15
  10. 2025Confounding Factors in Relating Model Performance to Morphologycited by 14
  11. 2024Two Counterexamples to Tokenization and the Noiseless Channelcited by 13
  12. 2021gaBERT -- an Irish Language Modelcited by 12
  13. 2023Effects of sub-word segmentation on performance of transformer language modelscited by 12
  14. 2024Transformers for Low-Resource Languages: Is F'eidir Linn!cited by 9
  15. 2025Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignmentcited by 9
  16. 2025MorphTok: Morphologically Grounded Tokenization for Indian Languagescited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Literal Token Overlap: Token Alignability for Multilinguality
  2. 2025The Art of Breaking Words: Rethinking Multilingual Tokenizer Design
  3. 2026How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them
  4. 2026TokEval: A Tokenizer Evaluation Suite
  5. 2025The Token Tax: Systematic Bias in Multilingual Tokenization
  6. 2026TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment
  7. 2026Separate Before You Compress: The WWHO Tokenization Architecture
  8. 2024Locally Measuring Cross-lingual Lexical Alignment: A Domain and Word Level Perspective