mlsanityData to 14 Oct 2026

Splintering Nonconcatenative Languages for Better Tokenization

arXiv 2503.14433 ↗First posted 18 Mar 2025cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Common subword tokenization algorithms like BPE and UnigramLM assume that text can be split into meaningful units by concatenative measures alone. This is not true for languages such as Hebrew and Arabic, where morphology is encoded in root-template patterns, or Malay and Georgian, where split affixes are common. We present SPLINTER, a pre-processing step which rearranges text into a linear form that better represents such nonconcatenative morphologies, enabling meaningful contiguous segments to be found by the tokenizer. We demonstrate SPLINTER's merit using both intrinsic measures evaluating token vocabularies in Hebrew, Arabic, and Malay; as well as on downstream tasks using BERT-architecture models trained for Hebrew.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 14 of its 29 references that are in the map. First posted on arXiv 18 Mar 2025.

Builds on

14 of 29 references.

References · 14

  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  2. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  3. 2024Tokenization Is More Than Compressioncited by 89
  4. 2023Tokenization and the Noiseless Channelcited by 59
  5. 2024Greed is All You Need: An Evaluation of Tokenizer Inference Methodscited by 34
  6. 2023Analyzing Cognitive Plausibility of Subword Tokenizationcited by 30
  7. 2020Neural Modeling for Named Entities and Morphology (NEMO^2)cited by 26
  8. 2023DictaBERT: A State-of-the-Art BERT Suite for Modern Hebrewcited by 17
  9. 2022Breaking Character: Are Subwords Good Enough for MRLs After All?cited by 13
  10. 2023HeRo: RoBERTa and Longformer Hebrew Language Modelscited by 12
  11. 2021Restoring Hebrew Diacritics Without a Dictionarycited by 10
  12. 2022A Second Wave of UD Hebrew Treebanking and Cross-Domain Parsingcited by 10
2 more
  1. 2024A Truly Joint Neural Architecture for Segmentation and Parsingcited by 4
  2. 2024MRL Parsing Without Tears: The Case of Hebrewcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025Evaluating Morphological Alignment of Tokenizers in 70 Languagescited by 22
  2. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 11
  3. 2025Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmeticcited by 1
  4. 2026Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMscited by 1
  5. 2025Hebrew Diacritics Restoration using Visual Representationcited by 0
  6. 2026When Similar Means Different: Evaluating LLMs on Arabic--Hebrew Cognatescited by 0