# Unnatural Error Correction: GPT-4 Can Almost Perfectly Handle Unnatural Scrambled Text

arXiv: https://arxiv.org/abs/2311.18805 · HTML: https://mlsanity.com/paper/2311.18805/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems](https://mlsanity.com/paper/1705.04146/) (2017)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [UL2: Unifying Language Learning Paradigms](https://mlsanity.com/paper/2205.05131/) (2022)
- [RealTime QA: What's the Answer Right Now?](https://mlsanity.com/paper/2207.13332/) (2022)
- [PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts](https://mlsanity.com/paper/2306.04528/) (2023)
- [On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective](https://mlsanity.com/paper/2302.12095/) (2023)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehension](https://mlsanity.com/paper/1902.00164/) (2019)
- [Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?](https://mlsanity.com/paper/2012.15180/) (2020)
- [UnNatural Language Inference](https://mlsanity.com/paper/2101.00010/) (2020)
- [Word Order Does Matter (And Shuffled Language Models Know It)](https://mlsanity.com/paper/2203.10995/) (2022)
- [An Error-Guided Correction Model for Chinese Spelling Error Correction](https://mlsanity.com/paper/2301.06323/) (2023)
- [Correcting the Autocorrect: Context-Aware Typographical Error Correction via Training Data Augmentation](https://mlsanity.com/paper/2005.01158/) (2020)

## Built on by (most cited first, 19 of 19)

- [Premise Order Matters in Reasoning with Large Language Models](https://mlsanity.com/paper/2402.08939/) (2024)
- [From Tokens to Words: On the Inner Lexicon of LLMs](https://mlsanity.com/paper/2410.05864/) (2024)
- [Tokenization Falling Short: On Subword Robustness in Large Language Models](https://mlsanity.com/paper/2406.11687/) (2024)
- [LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models](https://mlsanity.com/paper/2406.00605/) (2024)
- [Compressing Long Context for Enhancing RAG with AMR-based Concept Distillation](https://mlsanity.com/paper/2405.03085/) (2024)
- [Mitigating Reversal Curse in Large Language Models via Semantic-aware Permutation Training](https://mlsanity.com/paper/2403.00758/) (2024)
- [Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Ability](https://mlsanity.com/paper/2506.15629/) (2025)
- [EvolKV: Evolutionary KV Cache Compression for LLM Inference](https://mlsanity.com/paper/2509.08315/) (2025)
- [Did Translation Models Get More Robust Without Anyone Even Noticing?](https://mlsanity.com/paper/2403.03923/) (2024)
- [Word Form Matters: LLMs' Semantic Reconstruction under Typoglycemia](https://mlsanity.com/paper/2503.01714/) (2025)
- [LinFormer: A Linear-based Lightweight Transformer Architecture For Time-Aware MIMO Channel Prediction](https://mlsanity.com/paper/2410.21351/) (2024)
- [Mind Scramble: Unveiling Large Language Model Psychology Via Typoglycemia](https://mlsanity.com/paper/2410.01677/) (2024)
- [Understanding Subword Compositionality of Large Language Models](https://mlsanity.com/paper/2508.17953/) (2025)
- [Concept than Document: Context Compression via AMR-based Conceptual Entropy](https://mlsanity.com/paper/2511.18832/) (2025)
- [Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Models](https://mlsanity.com/paper/2607.21617/) (2026)
- [EXCGEC: A Benchmark for Edit-Wise Explainable Chinese Grammatical Error Correction](https://mlsanity.com/paper/2407.00924/) (2024)
- [Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Models](https://mlsanity.com/paper/2508.06621/) (2025)
- [Exploring the Limits of Large Language Models: A Systematic Evaluation of Masked Text Processing Ability through MskQA and MskCal](https://mlsanity.com/paper/2411.05665/) (2024)
- [Typoglycemia under the Hood: Investigating Language Models' Understanding of Scrambled Words](https://mlsanity.com/paper/2510.21326/) (2025)

## Abstract (arXiv)

While Large Language Models (LLMs) have achieved remarkable performance in many tasks, much about their inner workings remains unclear. In this study, we present novel experimental insights into the resilience of LLMs, particularly GPT-4, when subjected to extensive character-level permutations. To investigate this, we first propose the Scrambled Bench, a suite designed to measure the capacity of LLMs to handle scrambled input, in terms of both recovering scrambled sentences and answering questions given scrambled context. The experimental results indicate that most powerful LLMs demonstrate the capability akin to typoglycemia, a phenomenon where humans can understand the meaning of words even when the letters within those words are scrambled, as long as the first and last letters remain in place. More surprisingly, we found that only GPT-4 nearly flawlessly processes inputs with unnatural errors, even under the extreme condition, a task that poses significant challenges for other LLMs and often even for humans. Specifically, GPT-4 can almost perfectly reconstruct the original sentences from scrambled ones, decreasing the edit distance by 95%, even when all letters within each word are entirely scrambled. It is counter-intuitive that LLMs can exhibit such resilience despite severe disruption to input tokenization caused by scrambled text.