mlsanityData to 13 Oct 2026

EXECUTE: A Multilingual Benchmark for LLM Token Understanding

arXiv 2505.17784 ↗First posted 23 May 2025cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

The CUTE benchmark showed that LLMs struggle with character understanding in English. We extend it to more languages with diverse scripts and writing systems, introducing EXECUTE. Our simplified framework allows easy expansion to any language. Tests across multiple LLMs reveal that challenges in other languages are not always on the character level as in English. Some languages show word-level processing issues, some show no issues at all. We also examine sub-character tasks in Chinese, Japanese, and Korean to assess LLMs' understanding of character components.

Key facts. Cited by 4 papers in the map (data through 13 Oct 2026). Builds on 13 of its 17 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

13 of 17 references.

References · 13

  1. 2024The Llama 3 Herd of Modelscited by 14,486
  2. 2024Qwen2.5 Technical Reportcited by 4,840
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,376
  4. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 436
  5. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 141
  6. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 128
  7. 2023On the (In)Effectiveness of Large Language Models for Chinese Text Correctioncited by 54
  8. 2022What do tokens know about their characters and how do they know it?cited by 36
  9. 2024CUTE: Measuring LLMs' Understanding of Their Tokenscited by 30
  10. 2021Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokenscited by 26
  11. 2022Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Trainingcited by 16
  12. 2023Beyond English: Evaluating LLMs for Arabic Grammatical Error Correctioncited by 12
1 more
  1. 2024The Impact of Visual Information in Chinese Characters: Evaluating Large Models' Ability to Recognize and Utilize Radicalscited by 10

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 8
  2. 2026NOLLI: A Difficulty-Calibrated Puzzle Benchmark for Diagnosing the English-Korean Performance Gapcited by 0
  3. 2026Toppling the Hierarchy in Byte-level Language Modelingcited by 0
  4. 2026How Do Language Models Represent and Use Phonological Information for Allomorph Selection?cited by 0