# Hallucination Neurons and Where to Find Them: An Investigation into the existence of Hallucination Neurons

arXiv: https://arxiv.org/abs/2609.29781 · HTML: https://mlsanity.com/paper/2609.29781/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation
- LLM safety and security

## Builds on (most cited first, 20 of 20)

- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [MedGemma Technical Report](https://mlsanity.com/paper/2507.05201/) (2025)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Why Language Models Hallucinate](https://mlsanity.com/paper/2509.04664/) (2025)
- [Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models](https://mlsanity.com/paper/2411.14257/) (2024)
- [LLM Internal States Reveal Hallucination Risk Faced With a Query](https://mlsanity.com/paper/2407.03282/) (2024)
- [Finding Skill Neurons in Pre-trained Transformer-based Language Models](https://mlsanity.com/paper/2211.07349/) (2022)
- [What does the Knowledge Neuron Thesis Have to do with Knowledge?](https://mlsanity.com/paper/2405.02421/) (2024)
- [ICR Probe: Tracking Hidden State Dynamics for Reliable Hallucination Detection in LLMs](https://mlsanity.com/paper/2507.16488/) (2025)
- [Towards Understanding Multi-Task Learning (Generalization) of LLMs via Detecting and Exploring Task-Specific Neurons](https://mlsanity.com/paper/2407.06488/) (2024)
- [H-Neurons: On the Existence, Impact, and Origin of Hallucination-Associated Neurons in LLMs](https://mlsanity.com/paper/2512.01797/) (2025)
- [Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models](https://mlsanity.com/paper/2205.02023/) (2022)

## Abstract (arXiv)

Interpretable machine learning for Large Language Models (LLMs) increasingly relies on sparse probing methods that identify small sets of neurons claimed to detect and causally influence behaviors such as factuality recall, safety alignment, and hallucination. These claims have important implications for model auditing and behavioral steering, yet they are rarely tested against known failure modes of L1-regularized probing in correlated, high-dimensional feature spaces. We propose a five-step diagnostic protocol covering feature correlation, bootstrap stability, sparse versus dense ranking disagreement, intervention baselines, and cross-dataset evaluation as a minimum standard for sparse-neuron localization claims. We investigate prior work using our proposed approach, specifically on H-neurons using open-source LLMs across TriviaQA, BioASQ, and NQ-Open datasets. Our results demonstrate detection replicates across both models and datasets, and exceeds the original reported AUROC gaps for TriviaQA and BioASQ datasets. Gemma 3 4B consistently outperforms MedGemma 4B on matched datasets, with AUROC gaps of +0.311 versus +0.235 on TriviaQA, +0.474 versus +0.455 on BioASQ, and +0.128 versus +0.112 on NQ-Open respectively. Causal validation at n = 500 with five random seeds shows statistically significant effects beyond random same-layer baselines. At the same time, the diagnostic results indicate that the selected neurons are not uniquely localized. Across the three Gemma 3 4B settings, 19 of 22 selected H-Neurons have Pearson |r| > 0.7 with other features, bootstrap selections show only moderate stability, and sparse and dense rankings overlap only weakly. Our findings show that sparse predictive structure can coexist with non-unique neuron selection. Routine diagnostic validation is necessary to distinguish detection claims from localization claims in mechanistic interpretability.