mlsanityData to 14 Oct 2026

Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge

arXiv 2404.13292 ↗First posted 20 Apr 2024cs.CL cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

The popular subword tokenizers of current language models, such as Byte-Pair Encoding (BPE), are known not to respect morpheme boundaries, which affects the downstream performance of the models. While many improved tokenization algorithms have been proposed, their evaluation and cross-comparison is still an open problem. As a solution, we propose a combined intrinsic-extrinsic evaluation framework for subword tokenization. Intrinsic evaluation is based on our new UniMorph Labeller tool that classifies subword tokenization as either morphological or alien. Extrinsic evaluation, in turn, is performed via the Out-of-Vocabulary Generalization Challenge 1.0 benchmark, which consists of three newly specified downstream text classification tasks. Our empirical findings show that the accuracy of UniMorph Labeller is 98%, and that, in all language models studied (including ALBERT, BERT, RoBERTa, and DeBERTa), alien tokenization leads to poorer generalizations compared to morphological tokenization for semantic compositionality of word meanings.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 35 of its 57 references that are in the map. First posted on arXiv 20 Apr 2024.

Builds on

35 of 57 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  10. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  11. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  12. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
23 more
  1. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  2. 2019Compositionality decomposed: how do neural networks generalise?cited by 265
  3. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 192
  4. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 132
  5. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 115
  6. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  7. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  8. 2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61
  9. 2023Tokenization and the Noiseless Channelcited by 59
  10. 2018UniMorph 2.0: Universal Morphologycited by 42
  11. 2020Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translationcited by 38
  12. 2024Greed is All You Need: An Evaluation of Tokenizer Inference Methodscited by 34
  13. 2020An Empirical Study of Tokenization Strategies for Various Korean NLP Taskscited by 33
  14. 2023Analyzing Cognitive Plausibility of Subword Tokenizationcited by 30
  15. 2022The SIGMORPHON 2022 Shared Task on Morpheme Segmentationcited by 25
  16. 2020Mind Your Inflections! Improving NLP for Non-Standard Englishes with Base-Inflection Encodingcited by 22
  17. 2022Improving Tokenisation by Alternative Treatment of Spacescited by 19
  18. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  19. 2023Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship?cited by 13
  20. 2019On the Importance of Subword Information for Morphological Tasks in Truly Low-Resource Languagescited by 11
  21. 2022Tokenization Consistency Matters for Generative Models on Extractive NLP Taskscited by 7
  22. 2023CompoundPiece: Evaluating and Improving Decompounding Performance of Language Modelscited by 7
  23. 2024Using Contextual Information for Sentence-level Morpheme Segmentationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 1.

8 papers.

  1. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 48
  2. 2025Evaluating Morphological Alignment of Tokenizers in 70 Languagescited by 22
  3. 2024Token Erasure as a Footprint of Implicit Vocabulary Items in LLMscited by 18
  4. 2025Beyond Literal Token Overlap: Token Alignability for Multilingualitycited by 9
  5. 2025Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignmentcited by 9
  6. 2024Tokenization for Molecular Foundation Modelscited by 5
  7. 2025Rapid Word Learning Through Meta In-Context Learningcited by 4
  8. 2026Subword Segmental BabyLMs: Learning to Tokenise for Sample-Efficient Pretrainingcited by 0