# Investigating Neurons and Heads in Transformer-based LLMs for Typographical Errors

arXiv: https://arxiv.org/abs/2502.19669 · HTML: https://mlsanity.com/paper/2502.19669/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 27 Feb 2025.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [TextBugger: Generating Adversarial Text Against Real-world Applications](https://mlsanity.com/paper/1812.05271/) (2018)
- [Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers](https://mlsanity.com/paper/1801.04354/) (2018)
- [Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models](https://mlsanity.com/paper/2111.02840/) (2021)
- [PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts](https://mlsanity.com/paper/2306.04528/) (2023)
- [On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective](https://mlsanity.com/paper/2302.12095/) (2023)
- [Adaptively Sparse Transformers](https://mlsanity.com/paper/1909.00015/) (2019)
- [The Remarkable Robustness of LLMs: Stages of Inference?](https://mlsanity.com/paper/2406.19384/) (2024)
- [Identifying and Controlling Important Neurons in Neural Machine Translation](https://mlsanity.com/paper/1811.01157/) (2018)
- [Universal Neurons in GPT2 Language Models](https://mlsanity.com/paper/2401.12181/) (2024)
- [Information Flow Routes: Automatically Interpreting Language Models at Scale](https://mlsanity.com/paper/2403.00824/) (2024)
- [Successor Heads: Recurring, Interpretable Attention Heads In The Wild](https://mlsanity.com/paper/2312.09230/) (2023)
- [From Tokens to Words: On the Inner Lexicon of LLMs](https://mlsanity.com/paper/2410.05864/) (2024)
- [Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning](https://mlsanity.com/paper/2407.07011/) (2024)
- [On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex](https://mlsanity.com/paper/2301.12868/) (2023)
- [Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMs](https://mlsanity.com/paper/2406.09265/) (2024)
- [CUTE: Measuring LLMs' Understanding of Their Tokens](https://mlsanity.com/paper/2409.15452/) (2024)
- [Repetition Neurons: How Do Language Models Produce Repetitions?](https://mlsanity.com/paper/2410.13497/) (2024)
- [From Insights to Actions: The Impact of Interpretability and Analysis Research on NLP](https://mlsanity.com/paper/2406.12618/) (2024)
- [Mitigating the Language Mismatch and Repetition Issues in LLM-based Machine Translation via Model Editing](https://mlsanity.com/paper/2410.07054/) (2024)
- [How does GPT-2 Predict Acronyms? Extracting and Understanding a Circuit via Mechanistic Interpretability](https://mlsanity.com/paper/2405.04156/) (2024)
- [Detecting and Understanding Vulnerabilities in Language Models via Mechanistic Interpretability](https://mlsanity.com/paper/2407.19842/) (2024)
- [Noisy Exemplars Make Large Language Models More Robust: A Domain-Agnostic Behavioral Analysis](https://mlsanity.com/paper/2311.00258/) (2023)
- [Context-aware Stand-alone Neural Spelling Correction](https://mlsanity.com/paper/2011.06642/) (2020)
- [LEA: Improving Sentence Similarity Robustness to Typos Using Lexical Attention Bias](https://mlsanity.com/paper/2307.02912/) (2023)
- [Talking the Talk Does Not Entail Walking the Walk: On the Limits of Large Language Models in Lexical Entailment Recognition](https://mlsanity.com/paper/2406.14894/) (2024)
- [SubRegWeigh: Effective and Efficient Annotation Weighing with Subword Regularization](https://mlsanity.com/paper/2409.06216/) (2024)

## Abstract (arXiv)

This paper investigates how LLMs encode inputs with typos. We hypothesize that specific neurons and attention heads recognize typos and fix them internally using local and global contexts. We introduce a method to identify typo neurons and typo heads that work actively when inputs contain typos. Our experimental results suggest the following: 1) LLMs can fix typos with local contexts when the typo neurons in either the early or late layers are activated, even if those in the other are not. 2) Typo neurons in the middle layers are responsible for the core of typo-fixing with global contexts. 3) Typo heads fix typos by widely considering the context not focusing on specific tokens. 4) Typo neurons and typo heads work not only for typo-fixing but also for understanding general contexts.