mlsanityData to 14 Oct 2026

Tokenization Falling Short: On Subword Robustness in Large Language Models

arXiv 2406.11687 ↗First posted 17 Jun 2024cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

Language models typically tokenize raw text into sequences of subword identifiers from a predefined vocabulary, a process inherently sensitive to typographical errors, length variations, and largely oblivious to the internal structure of tokens--issues we term the curse of tokenization. In this study, we delve into these drawbacks and demonstrate that large language models (LLMs) remain susceptible to these problems. This study systematically investigates these challenges and their impact on LLMs through three critical research questions: (1) complex problem solving, (2) token structure probing, and (3) resilience to typographical variation. Our findings reveal that scaling model parameters can mitigate the issue of tokenization; however, LLMs still suffer from biases induced by typos and other text format variations. Our experiments show that subword regularization such as BPE-dropout can mitigate this issue. We release our evaluation code and data at https://github.com/FloatAI/TKEval.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 20 of its 35 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

20 of 35 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  9. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  10. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  11. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 232
  12. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 195
8 more
  1. 2016Exploring the Limits of Language Modelingcited by 173
  2. 2024HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalizationcited by 86
  3. 2020UnNatural Language Inferencecited by 69
  4. 2021HTLM: Hyper-Text Pre-Training and Prompting of Language Modelscited by 58
  5. 2022Word Order Does Matter (And Shuffled Language Models Know It)cited by 32
  6. 2022ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languagescited by 29
  7. 2023Unnatural Error Correction: GPT-4 Can Almost Perfectly Handle Unnatural Scrambled Textcited by 19
  8. 2024Autoregressive Pre-Training on Pixels and Textscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 12.

20 papers.

Sort:

By area:

  1. 2025Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizationscited by 17
  2. 2025Epistemological Fault Lines Between Human and Artificial Intelligencecited by 11
  3. 2025See the Text: From Tokenization to Visual Readingcited by 9
  4. 2025PixelWorld: How Far Are We from Perceiving Everything as Pixels?cited by 4
  5. 2025KORMo: Korean Open Reasoning Model for Everyonecited by 4
  6. 2025TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behaviorcited by 4
  7. 2025UTF-8 Plumbing: Byte-level Tokenizers Unavoidably Enable LLMs to Generate Ill-formed UTF-8cited by 3
  8. 2026Word Recovery in Large Language Models Enables Character-Level Tokenization Robustnesscited by 2
  9. 2026Decoupling semantics from vision: A framework for faithful visual-text compression evaluationcited by 2
  10. 2026TokEval: A Tokenizer Evaluation Suitecited by 2
  11. 2025ByteSpan: Information-Driven Subword Tokenisationcited by 1
  12. 2026Say Anything but This: When Tokenizer Betrays Reasoning in LLMscited by 1
8 more
  1. 2026Stochasticity in Tokenisation Improves Robustnesscited by 1
  2. 2026Effective Context in Transformers: An Analysis of Fragmentation and Tokenizationcited by 1
  3. 2026The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrievalcited by 0
  4. 2026Emergent retokenization symmetry in large language models: phenomenology and applicationscited by 0
  5. 2026Breaking Safety at the Token Boundary: How BPE Tokenization Creates Exploitable Gaps in LLM Alignmentcited by 0
  6. 2026MoirfEolas and Cr'iochScore: Developing Resources for and the Evaluation of Tokenization Alignment with Irish Morphologycited by 0
  7. 2026LS-AR: Future-Predictive Latent Steering in Autoregressive LLMscited by 0
  8. 2026What Is a Repeated Token Worth? The Scaling Geometry of Multi-Epoch Pretrainingcited by 0