mlsanityData to 7 Oct 2026

Morphological Typology in BPE Subword Productivity and Language Modeling

arXiv 2410.23656 ↗First posted 31 Oct 2024cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

This study investigates the impact of morphological typology on tokenization and language modeling performance. We focus on languages with synthetic and analytical morphological structures and examine their productivity when tokenized using the byte-pair encoding (BPE) algorithm. We compare the performance of models trained with similar amounts of data in different languages. Our experiments reveal that languages with synthetic features exhibit greater subword regularity and productivity with BPE tokenization and achieve better results in language modeling tasks. We also observe that the typological continuum from linguistic theory is reflected in several experiments. These findings suggest a correlation between morphological typology and BPE tokenization efficiency.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 14 of its 32 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

14 of 32 references.

References · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  4. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  5. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  6. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  7. 2022Impact of Tokenization on Language Models: An Analysis for Turkishcited by 62
  8. 2018Are All Languages Equally Hard to Language-Model?cited by 53
  9. 2023A Formal Perspective on Byte-Pair Encodingcited by 35
  10. 2018How Much Does Tokenization Affect Neural Machine Translation?cited by 26
  11. 2021The Effectiveness of Morphology-aware Segmentation in Low-Resource Neural Machine Translationcited by 16
  12. 2016Character-Level Language Modeling with Hierarchical Recurrent Neural Networkscited by 13
2 more
  1. 2017Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modelingcited by 8
  2. 2023Semantic Tokenizer for Enhanced Natural Language Processingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 10