# The Achilles' Heel of LLMs: How Altering a Handful of Neurons Can Cripple Language Abilities

arXiv: https://arxiv.org/abs/2510.10238 · HTML: https://mlsanity.com/paper/2510.10238/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 70 references that are in the map.
- First posted on arXiv 11 Oct 2025.

## Builds on (most cited first, 30 of 47)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [Towards Best Practices of Activation Patching in Language Models: Metrics and Methods](https://mlsanity.com/paper/2309.16042/) (2023)
- [Massive Activations in Large Language Models](https://mlsanity.com/paper/2402.17762/) (2024)
- [How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://mlsanity.com/paper/2305.00586/) (2023)
- [PMET: Precise Model Editing in a Transformer](https://mlsanity.com/paper/2308.08742/) (2023)
- [Understanding and Overcoming the Challenges of Efficient Transformer Quantization](https://mlsanity.com/paper/2109.12948/) (2021)
- [Why do LLMs attend to the first token?](https://mlsanity.com/paper/2504.02732/) (2025)
- [Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models](https://mlsanity.com/paper/2402.16438/) (2024)
- [BERT Busters: Outlier Dimensions that Disrupt Transformers](https://mlsanity.com/paper/2105.06990/) (2021)
- [HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization](https://mlsanity.com/paper/2402.16694/) (2024)
- [ReconBoost: Boosting Can Achieve Modality Reconcilement](https://mlsanity.com/paper/2405.09321/) (2024)
- [A Plug-and-Play Method for Controlled Text Generation](https://mlsanity.com/paper/2109.09707/) (2021)
- [Transformer Layers as Painters](https://mlsanity.com/paper/2407.09298/) (2024)
- [The Super Weight in Large Language Models](https://mlsanity.com/paper/2411.07191/) (2024)
- [Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challenges](https://mlsanity.com/paper/2409.02387/) (2024)
- [Vision Transformers Don't Need Trained Registers](https://mlsanity.com/paper/2506.08010/) (2025)
- [Runtime Monitoring Neuron Activation Patterns](https://mlsanity.com/paper/1809.06573/) (2018)
- [Contextual Feature Extraction Hierarchies Converge in Large Language Models and the Brain](https://mlsanity.com/paper/2401.17671/) (2024)
- [A Closer Look at Multimodal Representation Collapse](https://mlsanity.com/paper/2505.22483/) (2025)

## Built on by (most cited first, 7 of 7)

- [Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance](https://mlsanity.com/paper/2511.13254/) (2025)
- [Diffusion LLMs as Targets and Adversaries: Mechanistic Safety Exploits](https://mlsanity.com/paper/2608.07430/) (2026)
- [Sparse Neuron Ablation Triggers Catastrophic Collapse of the Language Core in Large Vision-Language Models](https://mlsanity.com/paper/2512.00918/) (2025)
- [Knowing Bias, Doing Better: Mitigating Social Bias in LLMs via Know-Bias Neuron Enhancement](https://mlsanity.com/paper/2601.21864/) (2026)
- [NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMs](https://mlsanity.com/paper/2608.08107/) (2026)
- [A Hyperbolicity Atlas of Large Language Model Hidden States](https://mlsanity.com/paper/2609.07053/) (2026)
- [Cascade: Hierarchical Recoverability Control for Large Language Model Unlearning](https://mlsanity.com/paper/2609.16890/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have become foundational tools in natural language processing, powering a wide range of applications and research. Many studies have shown that LLMs share significant similarities with the human brain. Recent neuroscience research has found that a small subset of biological neurons in the human brain are crucial for core cognitive functions, which raises a fundamental question: do LLMs also contain a small subset of critical neurons? In this paper, we investigate this question by proposing a Perturbation-based Causal Identification of Critical Neurons method to systematically locate such critical neurons in LLMs. Our findings reveal three key insights: (1) LLMs contain ultra-sparse critical neuron sets. Disrupting these critical neurons can cause a 72B-parameter model with over 1.1 billion neurons to completely collapse, with perplexity increasing by up to 20 orders of magnitude; (2) These critical neurons are not uniformly distributed, but tend to concentrate in the outer layers, particularly within the MLP downproj components; (3) Performance degradation exhibits sharp phase transitions, rather than a gradual decline, when these critical neurons are disrupted. Through comprehensive experiments across diverse model architectures and scales, we provide deeper analysis of these phenomena and their implications for LLM robustness and interpretability. These findings can offer guidance for developing more robust model architectures and improving deployment security in safety-critical applications. Our code is available at https://github.com/qqqqqqqzx/The-Achilles-Heel-of-LLMs.