mlsanityData to 7 Oct 2026

SubTokenTest: A Practical Benchmark for Real-World Sub-token Understanding

arXiv 2601.09089 ↗First posted 14 Jan 2026cs.CL cs.AIcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Recent advancements in large language models (LLMs) have significantly enhanced their reasoning capabilities. However, they continue to struggle with basic character-level tasks, such as counting letters in words, a problem rooted in their tokenization process. While existing benchmarks have highlighted this weakness through basic character operations, such failures are often dismissed due to lacking practical relevance. Yet, many real-world applications, such as navigating text-based maps or interpreting structured tables, rely heavily on precise sub-token understanding. In this regard, we introduce SubTokenTest, a comprehensive benchmark that assesses sub-token understanding through practical, utility-driven tasks. Our benchmark includes ten tasks across four domains and isolates tokenization-related failures by decoupling performance from complex reasoning. We provide a comprehensive evaluation of nine advanced LLMs. Additionally, we investigate the impact of test-time scaling on sub-token reasoning and explore how character-level information is encoded within the hidden states.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 19 of its 27 references that are in the map. First posted on arXiv 14 Jan 2026.

Builds on

19 of 27 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMs30 Apr
  2. 2025Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning25 Feb

Other references · 17

  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  3. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  4. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  5. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 157
  6. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 137
  7. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 112
  8. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 103
  9. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 46
  10. 2024Number Cookbook: Number Understanding of Language Models and How to Improve Itcited by 41
  11. 2021Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokenscited by 26
  12. 2024Case-Based or Rule-Based: How Do Transformers Do the Math?cited by 26
5 more
  1. 2022LMentry: A Language Model Benchmark of Elementary Language Taskscited by 21
  2. 2024Large Language Models Lack Understanding of Character Composition of Wordscited by 21
  3. 2025Beyond Single-Task: Robust Multi-Task Length Generalization for LLMscited by 7
  4. 2025StochasTok: Improving Fine-Grained Subword Understanding in LLMscited by 6
  5. 2025Spelling-out is not Straightforward: LLMs' Capability of Tokenization from Token to Characterscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks
  2. 2026SyntaxBench: A Statistical Diagnostic Framework for Character-Level Reasoning in Large Language Models
  3. 2025Tokenization Constraints in LLMs: A Study of Symbolic and Arithmetic Reasoning Limits
  4. 2026Say Anything but This: When Tokenizer Betrays Reasoning in LLMs
  5. 2025EfficientXLang: Towards Improving Token Efficiency Through Cross-Lingual Reasoning
  6. 2025OckBench: Measuring the Efficiency of LLM Reasoning
  7. 2025TASE: Token Awareness and Structured Evaluation for Multilingual Language Models
  8. 2026CorpusQA: A 10 Million Token Benchmark for Corpus-Level Analysis and Reasoning