mlsanityData to 7 Oct 2026

Analyzing Cognitive Plausibility of Subword Tokenization

arXiv 2310.13348 ↗First posted 20 Oct 2023cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

Subword tokenization has become the de-facto standard for tokenization, although comparative evaluations of subword vocabulary quality across languages are scarce. Existing evaluation studies focus on the effect of a tokenization algorithm on the performance in downstream tasks, or on engineering criteria such as the compression rate. We present a new evaluation paradigm that focuses on the cognitive plausibility of subword tokenization. We analyze the correlation of the tokenizer output with the response time and accuracy of human performance on a lexical decision task. We compare three tokenization algorithms across several languages and vocabulary sizes. Our results indicate that the UnigramLM algorithm yields less cognitively plausible tokenization behavior and a worse coverage of derivational morphemes, in contrast with prior work.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 15 of its 40 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

15 of 40 references.

References · 15

  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  4. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  5. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  6. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  7. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  8. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  9. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  10. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  11. 2020Improving Multilingual Models with Language-Clustered Vocabulariescited by 53
  12. 2020Morphology Matters: A Multilingual Language Modeling Analysiscited by 37
3 more
  1. 2022The SIGMORPHON 2022 Shared Task on Morpheme Segmentationcited by 25
  2. 2022BPE vs. Morphological Segmentation: A Case Study on Machine Translation of Four Polysynthetic Languagescited by 14
  3. 2021Wine is Not v i n. -- On the Compatibility of Tokenizations Across Languagescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 13; 2026: 7.

30 papers.

Sort:

By area:

  1. 2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61
  2. 2024Greed is All You Need: An Evaluation of Tokenizer Inference Methodscited by 34
  3. 2024BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Trainingcited by 30
  4. 2024The Foundations of Tokenization: Statistical and Computational Concernscited by 20
  5. 2025How Much is Enough? The Diminishing Returns of Tokenization Training Datacited by 15
  6. 2025AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptationcited by 15
  7. 2024On the Proper Treatment of Tokenization in Psycholinguisticscited by 12
  8. 2025Beyond Text Compression: Evaluating Tokenizers Across Scalescited by 11
  9. 2025Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignmentcited by 9
  10. 2024Rethinking Tokenization: Crafting Better Tokenizers for Large Language Modelscited by 8
  11. 2024Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challengecited by 8
  12. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 8
18 more
  1. 2024An Analysis of BPE Vocabulary Trimming in Neural Machine Translationcited by 7
  2. 2025Subword models struggle with word learning, but surprisal hides itcited by 6
  3. 2025Splintering Nonconcatenative Languages for Better Tokenizationcited by 6
  4. 2025IPA-CHILDES & G2P+: Feature-Rich Resources for Cross-Lingual Phonology and Phonemic Language Modelingcited by 6
  5. 2024Knowledge of Pretrained Language Models on Surface Information of Tokenscited by 5
  6. 2024Small Language Models Also Work With Small Vocabularies: Probing the Linguistic Abilities of Grapheme- and Phoneme-Based Baby Llamascited by 5
  7. 2026Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Modelscited by 5
  8. 2026Transducing Language Modelscited by 3
  9. 2025BabyLM's First Words: Word Segmentation as a Phonological Probing Taskcited by 2
  10. 2025Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluationcited by 2
  11. 2026SampoNLP: A Self-Referential Toolkit for Morphological Analysis of Subword Tokenizerscited by 2
  12. 2026On the Proper Treatment of Units in Surprisal Theorycited by 2
  13. 2026Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabetcited by 2
  14. 2025ByteSpan: Information-Driven Subword Tokenisationcited by 1
  15. 2025Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Modelscited by 1
  16. 2025Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibilitycited by 0
  17. 2026Vocabulary shapes cross-lingual variation of word-order learnability in language modelscited by 0
  18. 2026Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinyacited by 0