Investigating Neurons and Heads in Transformer-based LLMs for Typographical Errors
Not on the map. Why?
Abstract (arXiv)
This paper investigates how LLMs encode inputs with typos. We hypothesize that specific neurons and attention heads recognize typos and fix them internally using local and global contexts. We introduce a method to identify typo neurons and typo heads that work actively when inputs contain typos. Our experimental results suggest the following: 1) LLMs can fix typos with local contexts when the typo neurons in either the early or late layers are activated, even if those in the other are not. 2) Typo neurons in the middle layers are responsible for the core of typo-fixing with global contexts. 3) Typo heads fix typos by widely considering the context not focusing on specific tokens. 4) Typo neurons and typo heads work not only for typo-fixing but also for understanding general contexts.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 47 references that are in the map. First posted on arXiv 27 Feb 2025.
Builds on
30 of 47 references.
References · 30
- 2017Attention Is All You Needcited by 60,361
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
- 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
- 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
- 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 381
- 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 345
- 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 232
- 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 226
- 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 211
- 2019Adaptively Sparse Transformerscited by 155
- 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 146
- 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 111
18 more
- 2024Universal Neurons in GPT2 Language Modelscited by 102
- 2024Information Flow Routes: Automatically Interpreting Language Models at Scalecited by 95
- 2023Successor Heads: Recurring, Interpretable Attention Heads In The Wildcited by 77
- 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 48
- 2024Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learningcited by 46
- 2023On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codexcited by 41
- 2024Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMscited by 33
- 2024CUTE: Measuring LLMs' Understanding of Their Tokenscited by 31
- 2024Repetition Neurons: How Do Language Models Produce Repetitions?cited by 23
- 2024From Insights to Actions: The Impact of Interpretability and Analysis Research on NLPcited by 20
- 2024Mitigating the Language Mismatch and Repetition Issues in LLM-based Machine Translation via Model Editingcited by 18
- 2024How does GPT-2 Predict Acronyms? Extracting and Understanding a Circuit via Mechanistic Interpretabilitycited by 14
- 2024Detecting and Understanding Vulnerabilities in Language Models via Mechanistic Interpretabilitycited by 13
- 2023Noisy Exemplars Make Large Language Models More Robust: A Domain-Agnostic Behavioral Analysiscited by 6
- 2020Context-aware Stand-alone Neural Spelling Correctioncited by 4
- 2023LEA: Improving Sentence Similarity Robustness to Typos Using Lexical Attention Biascited by 2
- 2024Talking the Talk Does Not Entail Walking the Walk: On the Limits of Large Language Models in Lexical Entailment Recognitioncited by 1
- 2024SubRegWeigh: Effective and Efficient Annotation Weighing with Subword Regularizationcited by 1
Built on by
Not cited yet.