# Robustness of Large Language Models to Perturbations in Text

arXiv: https://arxiv.org/abs/2407.08989 · HTML: https://mlsanity.com/paper/2407.08989/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 52 references that are in the map.
- First posted on arXiv 12 Jul 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection](https://mlsanity.com/paper/2007.11464/) (2020)
- [A Simple Recipe for Multilingual Grammatical Error Correction](https://mlsanity.com/paper/2106.03830/) (2021)
- [JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction](https://mlsanity.com/paper/1702.04066/) (2017)
- [Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluation](https://mlsanity.com/paper/2304.01746/) (2023)
- [CoEdIT: Text Editing by Task-Specific Instruction Tuning](https://mlsanity.com/paper/2305.09857/) (2023)
- [Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change](https://mlsanity.com/paper/1606.02821/) (2016)
- [Survey of Computational Approaches to Lexical Semantic Change](https://mlsanity.com/paper/1811.06278/) (2018)
- [Synthetic Data Generation for Grammatical Error Correction with Tagged Corruption Models](https://mlsanity.com/paper/2105.13318/) (2021)
- [Temporal Analysis of Language through Neural Language Models](https://mlsanity.com/paper/1405.3515/) (2014)
- [Grammatical Error Correction: A Survey of the State of the Art](https://mlsanity.com/paper/2211.05166/) (2022)
- [Leveraging Contextual Embeddings for Detecting Diachronic Semantic Shift](https://mlsanity.com/paper/1912.01072/) (2019)
- [SynGEC: Syntax-Enhanced Grammatical Error Correction with a Tailored GEC-Oriented Parser](https://mlsanity.com/paper/2210.12484/) (2022)
- [Are Large Language Models Really Robust to Word-Level Perturbations?](https://mlsanity.com/paper/2309.11166/) (2023)
- [A Wind of Change: Detecting and Evaluating Lexical Semantic Change across Times and Domains](https://mlsanity.com/paper/1906.02979/) (2019)
- [On the Linearity of Semantic Change: Investigating Meaning Variation via Dynamic Graph Models](https://mlsanity.com/paper/1704.02497/) (2017)
- [HistBERT: A Pre-trained Language Model for Diachronic Lexical Semantic Analysis](https://mlsanity.com/paper/2202.03612/) (2022)
- [Towards Robustness to Label Noise in Text Classification via Noise Modeling](https://mlsanity.com/paper/2101.11214/) (2021)
- [Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understanding](https://mlsanity.com/paper/2104.06393/) (2021)
- [Understanding Model Robustness to User-generated Noisy Texts](https://mlsanity.com/paper/2110.07428/) (2021)
- [Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule](https://mlsanity.com/paper/2311.11813/) (2023)

## Built on by (most cited first, 11 of 11)

- [Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metrics](https://mlsanity.com/paper/2505.18658/) (2025)
- [Adversarial Vulnerabilities in Large Language Models for Time Series Forecasting](https://mlsanity.com/paper/2412.08099/) (2024)
- [Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbations](https://mlsanity.com/paper/2603.03332/) (2026)
- [Mind Scramble: Unveiling Large Language Model Psychology Via Typoglycemia](https://mlsanity.com/paper/2410.01677/) (2024)
- [Enabling AI Scientists to Recognize Innovation: A Domain-Agnostic Algorithm for Assessing Novelty](https://mlsanity.com/paper/2503.01508/) (2025)
- [Statistical Runtime Verification for LLMs via Robustness Estimation](https://mlsanity.com/paper/2504.17723/) (2025)
- [Measuring Reasoning Quality in LLMs: A Multi-Dimensional Behavioral Framework](https://mlsanity.com/paper/2605.24661/) (2026)
- [A framework for auditing grounding claims](https://mlsanity.com/paper/2512.06205/) (2025)
- [Beyond Confidence: The Rhythms of Reasoning in Generative Models](https://mlsanity.com/paper/2602.10816/) (2026)
- [The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrieval](https://mlsanity.com/paper/2605.07186/) (2026)
- [Spontaneous Context Restoration: How Language Models Recover from Corrupted Inputs](https://mlsanity.com/paper/2609.35475/) (2026)

## Abstract (arXiv)

Having a clean dataset has been the foundational assumption of most natural language processing (NLP) systems. However, properly written text is rarely found in real-world scenarios and hence, oftentimes invalidates the aforementioned foundational assumption. Recently, Large language models (LLMs) have shown impressive performance, but can they handle the inevitable noise in real-world data? This work tackles this critical question by investigating LLMs' resilience against morphological variations in text. To that end, we artificially introduce varying levels of noise into a diverse set of datasets and systematically evaluate LLMs' robustness against the corrupt variations of the original text. Our findings show that contrary to popular beliefs, generative LLMs are quiet robust to noisy perturbations in text. This is a departure from pre-trained models like BERT or RoBERTa whose performance has been shown to be sensitive to deteriorating noisy text. Additionally, we test LLMs' resilience on multiple real-world benchmarks that closely mimic commonly found errors in the wild. With minimal prompting, LLMs achieve a new state-of-the-art on the benchmark tasks of Grammar Error Correction (GEC) and Lexical Semantic Change (LSC). To empower future research, we also release a dataset annotated by humans stating their preference for LLM vs. human-corrected outputs along with the code to reproduce our results.