# Word Form Matters: LLMs' Semantic Reconstruction under Typoglycemia

arXiv: https://arxiv.org/abs/2503.01714 · HTML: https://mlsanity.com/paper/2503.01714/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 3 Mar 2025.

## Builds on (most cited first, 13 of 13)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [Do Large Language Models Latently Perform Multi-Hop Reasoning?](https://mlsanity.com/paper/2402.16837/) (2024)
- [Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?](https://mlsanity.com/paper/2012.15180/) (2020)
- [BERT & Family Eat Word Salad: Experiments with Text Understanding](https://mlsanity.com/paper/2101.03453/) (2021)
- [Unnatural Error Correction: GPT-4 Can Almost Perfectly Handle Unnatural Scrambled Text](https://mlsanity.com/paper/2311.18805/) (2023)
- [WinoDict: Probing language models for in-context word acquisition](https://mlsanity.com/paper/2209.12153/) (2022)
- [Word Importance Explains How Prompts Affect Language Model Outputs](https://mlsanity.com/paper/2403.03028/) (2024)
- [Mind Scramble: Unveiling Large Language Model Psychology Via Typoglycemia](https://mlsanity.com/paper/2410.01677/) (2024)
- [What Kinds of Tokens Benefit from Distant Text? An Analysis on Long Context Language Modeling](https://mlsanity.com/paper/2406.11238/) (2024)

## Built on by (most cited first, 8 of 8)

- [Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learning](https://mlsanity.com/paper/2505.12477/) (2025)
- [ManipLVM-R1: Reinforcement Learning for Reasoning in Embodied Manipulation with Large Vision-Language Models](https://mlsanity.com/paper/2505.16517/) (2025)
- [DyFlow: Dynamic Workflow Framework for Agentic Reasoning](https://mlsanity.com/paper/2509.26062/) (2025)
- [Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Models](https://mlsanity.com/paper/2505.15406/) (2025)
- [Spelling-out is not Straightforward: LLMs' Capability of Tokenization from Token to Characters](https://mlsanity.com/paper/2506.10641/) (2025)
- [Measure, Don't Optimize: Forecasting Recovery in LLM Unlearning](https://mlsanity.com/paper/2608.11408/) (2026)
- [Typoglycemia under the Hood: Investigating Language Models' Understanding of Scrambled Words](https://mlsanity.com/paper/2510.21326/) (2025)
- [GTA: Graph Theory Agent and Benchmark for Algorithmic Graph Reasoning with LLMs](https://mlsanity.com/paper/2609.12265/) (2026)

## Abstract (arXiv)

Human readers can efficiently comprehend scrambled words, a phenomenon known as Typoglycemia, primarily by relying on word form; if word form alone is insufficient, they further utilize contextual cues for interpretation. While advanced large language models (LLMs) exhibit similar abilities, the underlying mechanisms remain unclear. To investigate this, we conduct controlled experiments to analyze the roles of word form and contextual information in semantic reconstruction and examine LLM attention patterns. Specifically, we first propose SemRecScore, a reliable metric to quantify the degree of semantic reconstruction, and validate its effectiveness. Using this metric, we study how word form and contextual information influence LLMs' semantic reconstruction ability, identifying word form as the core factor in this process. Furthermore, we analyze how LLMs utilize word form and find that they rely on specialized attention heads to extract and process word form information, with this mechanism remaining stable across varying levels of word scrambling. This distinction between LLMs' fixed attention patterns primarily focused on word form and human readers' adaptive strategy in balancing word form and contextual information provides insights into enhancing LLM performance by incorporating human-like, context-aware mechanisms.