# Large Language Models Lack Understanding of Character Composition of Words

arXiv: https://arxiv.org/abs/2405.11357 · HTML: https://mlsanity.com/paper/2405.11357/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 18 May 2024.

## Builds on (most cited first, 15 of 15)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Character-Aware Neural Language Models](https://mlsanity.com/paper/1508.06615/) (2015)
- [SVTR: Scene Text Recognition with a Single Visual Model](https://mlsanity.com/paper/2205.00159/) (2022)
- [Language models are not naysayers: An analysis of language models on negation benchmarks](https://mlsanity.com/paper/2306.08189/) (2023)
- [Limitations of Language Models in Arithmetic and Symbolic Induction](https://mlsanity.com/paper/2208.05051/) (2022)
- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [LMentry: A Language Model Benchmark of Elementary Language Tasks](https://mlsanity.com/paper/2211.02069/) (2022)
- [Charagram: Embedding Words and Sentences via Character n-grams](https://mlsanity.com/paper/1607.02789/) (2016)
- [Language Models Don't Learn the Physical Manifestation of Language](https://mlsanity.com/paper/2402.11349/) (2024)
- [Large Language Models for Simultaneous Named Entity Extraction and Spelling Correction](https://mlsanity.com/paper/2403.00528/) (2024)

## Built on by (most cited first, 22 of 22)

- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language Models](https://mlsanity.com/paper/2505.14172/) (2025)
- [From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning](https://mlsanity.com/paper/2407.00902/) (2024)
- [StochasTok: Improving Fine-Grained Subword Understanding in LLMs](https://mlsanity.com/paper/2506.01687/) (2025)
- [Subword models struggle with word learning, but surprisal hides it](https://mlsanity.com/paper/2502.12835/) (2025)
- [Measuring Progress Toward AGI: A Cognitive Framework](https://mlsanity.com/paper/2605.28405/) (2026)
- [Spelling-out is not Straightforward: LLMs' Capability of Tokenization from Token to Characters](https://mlsanity.com/paper/2506.10641/) (2025)
- [CharBench: Evaluating the Role of Tokenization in Character-Level Tasks](https://mlsanity.com/paper/2508.02591/) (2025)
- [Prompt-Based One-Shot Exact Length-Controlled Generation with LLMs](https://mlsanity.com/paper/2508.13805/) (2025)
- [Enhancing Character-Level Understanding in LLMs through Token Internal Structure Learning](https://mlsanity.com/paper/2411.17679/) (2024)
- [Multilingual LLMs Struggle to Link Orthography and Semantics in Bilingual Word Processing](https://mlsanity.com/paper/2501.09127/) (2025)
- [ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs](https://mlsanity.com/paper/2603.26516/) (2026)
- [From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks](https://mlsanity.com/paper/2604.00778/) (2026)
- [StringLLM: Understanding the String Processing Capability of Large Language Models](https://mlsanity.com/paper/2410.01208/) (2024)
- [Aligning Generalisation Between Humans and Machines](https://mlsanity.com/paper/2411.15626/) (2024)
- [ChaI-TeA: A Benchmark for Evaluating Autocompletion of Interactions with LLM-based Chatbots](https://mlsanity.com/paper/2412.18377/) (2024)
- [Do multimodal models imagine electric sheep?](https://mlsanity.com/paper/2605.09693/) (2026)
- [Training Language Models with homotokens Leads to Delayed Overfitting](https://mlsanity.com/paper/2601.02867/) (2026)
- [SubTokenTest: A Practical Benchmark for Real-World Sub-token Understanding](https://mlsanity.com/paper/2601.09089/) (2026)
- [Spelling Bee Embeddings for Language Modeling](https://mlsanity.com/paper/2601.18030/) (2026)
- [Beyond Accuracy: Diagnosing Algebraic Reasoning Failures in LLMs Across Nine Complexity Dimensions](https://mlsanity.com/paper/2604.06799/) (2026)
- [SCRIPTIOC-BENCH: A Benchmark for Recognizing Actionable Threat Intelligence from Script-Based Malware using LLMs](https://mlsanity.com/paper/2609.06149/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable performances on a wide range of natural language tasks. Yet, LLMs' successes have been largely restricted to tasks concerning words, sentences, or documents, and it remains questionable how much they understand the minimal units of text, namely characters. In this paper, we examine contemporary LLMs regarding their ability to understand character composition of words, and show that most of them fail to reliably carry out even the simple tasks that can be handled by humans with perfection. We analyze their behaviors with comparison to token level performances, and discuss the potential directions for future research.