mlsanityData to 14 Oct 2026

Reasoning Over the Glyphs: Evaluation of LLM's Decipherment of Rare Scripts

arXiv 2501.17785 ↗First posted 29 Jan 2025cs.CL cs.LGcited by 4

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We explore the capabilities of LVLMs and LLMs in deciphering rare scripts not encoded in Unicode. We introduce a novel approach to construct a multimodal dataset of linguistic puzzles involving such scripts, utilizing a tokenization method for language glyphs. Our methods include the Picture Method for LVLMs and the Description Method for LLMs, enabling these models to tackle these challenges. We conduct experiments using prominent models, GPT-4o, Gemini, and Claude 3.5 Sonnet, on linguistic puzzles. Our findings reveal the strengths and limitations of current AI methods in linguistic decipherment, highlighting the impact of Unicode encoding on model performance and the challenges of modeling visual language tokens through descriptions. Our study advances understanding of AI's potential in linguistic decipherment and underscores the need for further research.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 7 of its 12 references that are in the map. First posted on arXiv 29 Jan 2025.

Builds on

7 of 12 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages10 Jun

Other references · 6

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  3. 2024An open dataset for oracle bone script recognition and deciphermentcited by 19
  4. 2024Deciphering Oracle Bone Language with Diffusion Modelscited by 18
  5. 2019Neural Decipherment via Minimum-Cost Flow: from Ugaritic to Linear Bcited by 7
  6. 2023Bridging the Gap: Deciphering Tabular Data Using Large Language Modelcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Mitigating Long-tail Distribution in Oracle Bone Inscriptions: Dataset, Model, and Benchmarkcited by 5
  2. 2025LingBench++: A Linguistically-Informed Benchmark and Reasoning Framework for Multi-Step and Cross-Cultural Inference with LLMscited by 5
  3. 2025UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?cited by 3
  4. 2026Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branchescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond 'One Language, One Script': Quantifying Orthographic Bias in Multilingual VLMs with PuMVR
  2. 2024REBUS: A Robust Evaluation Benchmark of Understanding Symbols
  3. 2024Impact of Non-Standard Unicode Characters on Security and Comprehension in Large Language Models
  4. 2025RomanLens: The Role Of Latent Romanization In Multilinguality In LLMs
  5. 2024modeLing: A Novel Dataset for Testing Linguistic Reasoning in Language Models
  6. 2025Unnatural Languages Are Not Bugs but Features for LLMs
  7. 2025Evaluating Robustness of Large Language Models Against Multilingual Typographical Errors
  8. 2024Picturing Ambiguity: A Visual Twist on the Winograd Schema Challenge