mlsanityData to 7 Oct 2026

Greed is All You Need: An Evaluation of Tokenizer Inference Methods

arXiv 2403.01289 ↗First posted 2 Mar 2024cs.CLcited by 34

Not on the map. Why?

Abstract (arXiv)

While subword tokenizers such as BPE and WordPiece are typically used to build vocabularies for NLP models, the method of decoding text into a sequence of tokens from these vocabularies is often left unspecified, or ill-suited to the method in which they were constructed. We provide a controlled analysis of seven tokenizer inference methods across four different algorithms and three vocabulary sizes, performed on a novel intrinsic evaluation suite we curated for English, combining measures rooted in morphology, cognition, and information theory. We show that for the most commonly used tokenizers, greedy inference performs surprisingly well; and that SaGe, a recently-introduced contextually-informed tokenizer, outperforms all others on morphological alignment.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 22 of its 43 references that are in the map. First posted on arXiv 2 Mar 2024.

Builds on

22 of 43 references.

References · 22

  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  2. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  3. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  4. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  5. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  6. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  7. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  8. 2024Tokenization Is More Than Compressioncited by 87
  9. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 74
  10. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  11. 2023Tokenization and the Noiseless Channelcited by 58
  12. 2019Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimickingcited by 40
10 more
  1. 2020Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translationcited by 38
  2. 2023Analyzing Cognitive Plausibility of Subword Tokenizationcited by 30
  3. 2022Improving Tokenisation by Alternative Treatment of Spacescited by 19
  4. 2023Tokenization Matters: Navigating Data-Scarce Tokenization for Gender Inclusive Language Technologiescited by 16
  5. 2022BPE vs. Morphological Segmentation: A Case Study on Machine Translation of Four Polysynthetic Languagescited by 14
  6. 2020Will it Unblend?cited by 11
  7. 2023What changes when you randomly choose BPE merge operations? Not muchcited by 10
  8. 2023CompoundPiece: Evaluating and Improving Decompounding Performance of Language Modelscited by 7
  9. 2019Character Eyes: Seeing Language through Character-Level Taggerscited by 3
  10. 2024Word Boundary Information Isn't Useful for Encoder Language Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 13; 2026: 11.

34 papers.

Sort:

  1. 2024Why do language models perform worse for morphologically complex languages?cited by 62
  2. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 40
  3. 2024Zero-Shot Tokenizer Transfercited by 30
  4. 2025Evaluating Morphological Alignment of Tokenizers in 70 Languagescited by 22
  5. 2024The Foundations of Tokenization: Statistical and Computational Concernscited by 20
  6. 2024Retrofitting Large Language Models with Dynamic Tokenizationcited by 15
  7. 2025How Much is Enough? The Diminishing Returns of Tokenization Training Datacited by 15
  8. 2025AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptationcited by 15
  9. 2025Tokenization is Sensitive to Language Variationcited by 12
  10. 2025Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignmentcited by 9
  11. 2024Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challengecited by 8
  12. 2024Hit the Sweet Spot! Span-Level Ensemble for Large Language Modelscited by 8
22 more
  1. 2024When Every Token Counts: Optimal Segmentation for Low-Resource Language Modelscited by 8
  2. 2024An Analysis of BPE Vocabulary Trimming in Neural Machine Translationcited by 7
  3. 2025Subword models struggle with word learning, but surprisal hides itcited by 6
  4. 2025Splintering Nonconcatenative Languages for Better Tokenizationcited by 6
  5. 2024Different Tokenization Schemes Lead to Comparable Performance in Spanish Number Agreementcited by 5
  6. 2024Small Language Models Also Work With Small Vocabularies: Probing the Linguistic Abilities of Grapheme- and Phoneme-Based Baby Llamascited by 5
  7. 2025Tokenisation over Bounded Alphabets is Hardcited by 5
  8. 2026Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Modelscited by 5
  9. 2026MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignmentcited by 5
  10. 2025CharBench: Evaluating the Role of Tokenization in Character-Level Taskscited by 3
  11. 2026Tokenization with Split Treescited by 2
  12. 2025ByteSpan: Information-Driven Subword Tokenisationcited by 1
  13. 2025Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Modelscited by 1
  14. 2026Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Featurescited by 1
  15. 2026Joint Optimization for Greedy Longest-match Tokenizationcited by 1
  16. 2026Pruned BPE: Post-training Visibility Pruning and Token Reallocation for Byte Pair Encodingcited by 1
  17. 2025TreeRanker: Fast and Model-agnostic Ranking System for Code Suggestions in IDEscited by 0
  18. 2026How Long Is a Piece of String? A Brief Empirical Analysis of Tokenizerscited by 0
  19. 2026What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling?cited by 0
  20. 2026Dynamic-Programming-Guided Hierarchical BPE and Empirical Analysis of Vocabulary Pruningcited by 0
  21. 2026Fewer Words, Not Fewer Tokens: Measuring the Sanskrit Tokenization Penalty per Propositioncited by 0
  22. 2026Counting and Min-Cost Encoding for Tokenization in Large Language Modelscited by 0