# CUTE: Measuring LLMs' Understanding of Their Tokens

arXiv: https://arxiv.org/abs/2409.15452 · HTML: https://mlsanity.com/paper/2409.15452/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 23 Sep 2024.

## Builds on (most cited first, 16 of 16)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [Charformer: Fast Character Transformers via Gradient-based Subword Tokenization](https://mlsanity.com/paper/2106.12672/) (2021)
- [Aya 23: Open Weight Releases to Further Multilingual Progress](https://mlsanity.com/paper/2405.15032/) (2024)
- [Fully Character-Level Neural Machine Translation without Explicit Segmentation](https://mlsanity.com/paper/1610.03017/) (2016)
- [Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4](https://mlsanity.com/paper/2312.16171/) (2023)
- [What do tokens know about their characters and how do they know it?](https://mlsanity.com/paper/2206.02608/) (2022)
- [Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens](https://mlsanity.com/paper/2108.11193/) (2021)
- [Why don't people use character-level machine translation?](https://mlsanity.com/paper/2110.08191/) (2021)
- [Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Training](https://mlsanity.com/paper/2212.09897/) (2022)
- [Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation](https://mlsanity.com/paper/2302.14220/) (2023)
- [Subword-Delimited Downsampling for Better Character-Level Translation](https://mlsanity.com/paper/2212.01304/) (2022)

## Built on by (most cited first, 30 of 31)

- [SuperBPE: Space Travel for Language Models](https://mlsanity.com/paper/2503.13423/) (2025)
- [Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations](https://mlsanity.com/paper/2506.19004/) (2025)
- [From Bytes to Ideas: Language Modeling with Autoregressive U-Nets](https://mlsanity.com/paper/2506.14761/) (2025)
- [Bolmo: Byteifying the Next Generation of Language Models](https://mlsanity.com/paper/2512.15586/) (2025)
- [StochasTok: Improving Fine-Grained Subword Understanding in LLMs](https://mlsanity.com/paper/2506.01687/) (2025)
- [Speculating LLMs' Chinese Training Data Pollution from Their Tokens](https://mlsanity.com/paper/2508.17771/) (2025)
- [EXECUTE: A Multilingual Benchmark for LLM Token Understanding](https://mlsanity.com/paper/2505.17784/) (2025)
- [Spelling-out is not Straightforward: LLMs' Capability of Tokenization from Token to Characters](https://mlsanity.com/paper/2506.10641/) (2025)
- [Learning Composable Chains-of-Thought](https://mlsanity.com/paper/2505.22635/) (2025)
- [From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks](https://mlsanity.com/paper/2604.00778/) (2026)
- [Fast Byte Latent Transformer](https://mlsanity.com/paper/2605.08044/) (2026)
- [Strategies for Span Labeling with Large Language Models](https://mlsanity.com/paper/2601.16946/) (2026)
- [Word Recovery in Large Language Models Enables Character-Level Tokenization Robustness](https://mlsanity.com/paper/2603.10771/) (2026)
- [Enhancing LLM Character-Level Manipulation via Divide and Conquer](https://mlsanity.com/paper/2502.08180/) (2025)
- [A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents](https://mlsanity.com/paper/2602.08964/) (2026)
- [ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer](https://mlsanity.com/paper/2603.03583/) (2026)
- [Stochasticity in Tokenisation Improves Robustness](https://mlsanity.com/paper/2604.16037/) (2026)
- [AoA: Theorem Proving Agent over Abstract Syntax Tree of Redesigned Language](https://mlsanity.com/paper/2607.16372/) (2026)
- [Investigating Neurons and Heads in Transformer-based LLMs for Typographical Errors](https://mlsanity.com/paper/2502.19669/) (2025)
- [TASE: Token Awareness and Structured Evaluation for Multilingual Language Models](https://mlsanity.com/paper/2508.05468/) (2025)
- [Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Models](https://mlsanity.com/paper/2511.21086/) (2025)
- [How Do Language Models Acquire Character-Level Information?](https://mlsanity.com/paper/2602.05347/) (2026)
- [PACUTE: Phonology-, Affix-, and Character-level Understanding of Tokens for Filipino](https://mlsanity.com/paper/2606.15144/) (2026)
- [Emergent retokenization symmetry in large language models: phenomenology and applications](https://mlsanity.com/paper/2606.15521/) (2026)
- [NOLLI: A Difficulty-Calibrated Puzzle Benchmark for Diagnosing the English-Korean Performance Gap](https://mlsanity.com/paper/2608.04397/) (2026)
- [Toppling the Hierarchy in Byte-level Language Modeling](https://mlsanity.com/paper/2609.00463/) (2026)
- [How Do Language Models Represent and Use Phonological Information for Allomorph Selection?](https://mlsanity.com/paper/2609.04708/) (2026)
- [Playing log(N)-Questions over Wikipedia Abstracts: How Per-Round Errors Compound Under Information Asymmetry](https://mlsanity.com/paper/2609.19113/) (2026)
- [More Than Words: Compositional Tokenization for Efficient Language Models](https://mlsanity.com/paper/2610.05597/) (2026)
- [Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation](https://mlsanity.com/paper/2610.05978/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) show remarkable performance on a wide variety of tasks. Most LLMs split text into multi-character tokens and process them as atomic units without direct access to individual characters. This raises the question: To what extent can LLMs learn orthographic information? To answer this, we propose a new benchmark, CUTE, which features a collection of tasks designed to test the orthographic knowledge of LLMs. We evaluate popular LLMs on CUTE, finding that most of them seem to know the spelling of their tokens, yet fail to use this information effectively to manipulate text, calling into question how much of this knowledge is generalizable.