mlsanityData to 14 Oct 2026

Large Language Models Lack Understanding of Character Composition of Words

arXiv 2405.11357 ↗First posted 18 May 2024cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable performances on a wide range of natural language tasks. Yet, LLMs' successes have been largely restricted to tasks concerning words, sentences, or documents, and it remains questionable how much they understand the minimal units of text, namely characters. In this paper, we examine contemporary LLMs regarding their ability to understand character composition of words, and show that most of them fail to reliably carry out even the simple tasks that can be handled by humans with perfection. We analyze their behaviors with comparison to token level performances, and discuss the potential directions for future research.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 15 of its 24 references that are in the map. First posted on arXiv 18 May 2024.

Builds on

15 of 24 references.

References · 15

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  4. 2018Deep contextualized word representationscited by 2,290
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  6. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  7. 2015Character-Aware Neural Language Modelscited by 132
  8. 2022SVTR: Scene Text Recognition with a Single Visual Modelcited by 88
  9. 2023Language models are not naysayers: An analysis of language models on negation benchmarkscited by 74
  10. 2022Limitations of Language Models in Arithmetic and Symbolic Inductioncited by 71
  11. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  12. 2022LMentry: A Language Model Benchmark of Elementary Language Taskscited by 22
3 more
  1. 2016Charagram: Embedding Words and Sentences via Character n-gramscited by 14
  2. 2024Language Models Don't Learn the Physical Manifestation of Languagecited by 4
  3. 2024Large Language Models for Simultaneous Named Entity Extraction and Spelling Correctioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 7; 2026: 10.

22 papers.

Sort:

By area:

  1. 2026Large Language Model Reasoning Failurescited by 21
  2. 2024From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learningcited by 11
  3. 2025The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language Modelscited by 11
  4. 2025StochasTok: Improving Fine-Grained Subword Understanding in LLMscited by 7
  5. 2025Subword models struggle with word learning, but surprisal hides itcited by 6
  6. 2026Measuring Progress Toward AGI: A Cognitive Frameworkcited by 6
  7. 2025Spelling-out is not Straightforward: LLMs' Capability of Tokenization from Token to Characterscited by 4
  8. 2025Prompt-Based One-Shot Exact Length-Controlled Generation with LLMscited by 4
  9. 2024Enhancing Character-Level Understanding in LLMs through Token Internal Structure Learningcited by 3
  10. 2025Multilingual LLMs Struggle to Link Orthography and Semantics in Bilingual Word Processingcited by 3
  11. 2025CharBench: Evaluating the Role of Tokenization in Character-Level Taskscited by 3
  12. 2026ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMscited by 3
10 more
  1. 2026From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Taskscited by 3
  2. 2024StringLLM: Understanding the String Processing Capability of Large Language Modelscited by 2
  3. 2024Aligning Generalisation Between Humans and Machinescited by 2
  4. 2024ChaI-TeA: A Benchmark for Evaluating Autocompletion of Interactions with LLM-based Chatbotscited by 1
  5. 2026Do multimodal models imagine electric sheep?cited by 1
  6. 2026Training Language Models with homotokens Leads to Delayed Overfittingcited by 0
  7. 2026SubTokenTest: A Practical Benchmark for Real-World Sub-token Understandingcited by 0
  8. 2026Spelling Bee Embeddings for Language Modelingcited by 0
  9. 2026Beyond Accuracy: Diagnosing Algebraic Reasoning Failures in LLMs Across Nine Complexity Dimensionscited by 0
  10. 2026SCRIPTIOC-BENCH: A Benchmark for Recognizing Actionable Threat Intelligence from Script-Based Malware using LLMscited by 0