mlsanityData to 7 Oct 2026

CUTE: Measuring LLMs' Understanding of Their Tokens

arXiv 2409.15452 ↗First posted 23 Sep 2024cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) show remarkable performance on a wide variety of tasks. Most LLMs split text into multi-character tokens and process them as atomic units without direct access to individual characters. This raises the question: To what extent can LLMs learn orthographic information? To answer this, we propose a new benchmark, CUTE, which features a collection of tasks designed to test the orthographic knowledge of LLMs. We evaluate popular LLMs on CUTE, finding that most of them seem to know the spelling of their tokens, yet fail to use this information effectively to manipulate text, calling into question how much of this knowledge is generalizable.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 16 of its 23 references that are in the map. First posted on arXiv 23 Sep 2024.

Builds on

16 of 23 references.

References · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  4. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  5. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  6. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  7. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  8. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 121
  9. 2016Fully Character-Level Neural Machine Translation without Explicit Segmentationcited by 55
  10. 2023Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4cited by 48
  11. 2022What do tokens know about their characters and how do they know it?cited by 36
  12. 2021Why don't people use character-level machine translation?cited by 27
4 more
  1. 2021Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokenscited by 26
  2. 2022Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Trainingcited by 16
  3. 2023Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translationcited by 14
  4. 2022Subword-Delimited Downsampling for Better Character-Level Translationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 15.

28 papers.

Sort:

By area:

  1. 2025SuperBPE: Space Travel for Language Modelscited by 58
  2. 2025Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizationscited by 16
  3. 2025From Bytes to Ideas: Language Modeling with Autoregressive U-Netscited by 15
  4. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 8
  5. 2025StochasTok: Improving Fine-Grained Subword Understanding in LLMscited by 6
  6. 2025Speculating LLMs' Chinese Training Data Pollution from Their Tokenscited by 5
  7. 2025EXECUTE: A Multilingual Benchmark for LLM Token Understandingcited by 4
  8. 2025Learning Composable Chains-of-Thoughtcited by 3
  9. 2025Spelling-out is not Straightforward: LLMs' Capability of Tokenization from Token to Characterscited by 3
  10. 2026From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Taskscited by 3
  11. 2026Strategies for Span Labeling with Large Language Modelscited by 2
  12. 2026Word Recovery in Large Language Models Enables Character-Level Tokenization Robustnesscited by 2
16 more
  1. 2026Fast Byte Latent Transformercited by 2
  2. 2025Enhancing LLM Character-Level Manipulation via Divide and Conquercited by 1
  3. 2026Stochasticity in Tokenisation Improves Robustnesscited by 1
  4. 2026AoA: Theorem Proving Agent over Abstract Syntax Tree of Redesigned Languagecited by 1
  5. 2025Investigating Neurons and Heads in Transformer-based LLMs for Typographical Errorscited by 0
  6. 2025TASE: Token Awareness and Structured Evaluation for Multilingual Language Modelscited by 0
  7. 2025Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Modelscited by 0
  8. 2026How Do Language Models Acquire Character-Level Information?cited by 0
  9. 2026A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agentscited by 0
  10. 2026ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizercited by 0
  11. 2026PACUTE: Phonology-, Affix-, and Character-level Understanding of Tokens for Filipinocited by 0
  12. 2026Emergent retokenization symmetry in large language models: phenomenology and applicationscited by 0
  13. 2026NOLLI: A Difficulty-Calibrated Puzzle Benchmark for Diagnosing the English-Korean Performance Gapcited by 0
  14. 2026Toppling the Hierarchy in Byte-level Language Modelingcited by 0
  15. 2026How Do Language Models Represent and Use Phonological Information for Allomorph Selection?cited by 0
  16. 2026Playing log(N)-Questions over Wikipedia Abstracts: How Per-Round Errors Compound Under Information Asymmetrycited by 0