# Hallucination Is Linearly Decodable from Mid-Layer Hidden States in Quantized LLMs

arXiv: https://arxiv.org/abs/2606.02628 · HTML: https://mlsanity.com/paper/2606.02628/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination detection from model internals, one of 13 directions in Mechanistic interpretability.
- Builds on 20 of its 23 references that are in the map.
- First posted on arXiv 30 May 2026.

## Placed in

- Mechanistic interpretability / [Hallucination detection from model internals](https://mlsanity.com/mechinterp/hallucination-detection-from-model-internals/)

## Builds on (most cited first, 20 of 20)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2305.11747/) (2023)
- [INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection](https://mlsanity.com/paper/2402.03744/) (2024)
- [LM vs LM: Detecting Factual Errors via Cross Examination](https://mlsanity.com/paper/2305.13281/) (2023)

## Built on by (most cited first, 3 of 3)

- [Understanding Why Language Models Hallucinate: Testing Reasoning Against Priors](https://mlsanity.com/paper/2607.00447/) (2026)
- [Confidently Wrong: Detecting Hallucinations in Financial Question Answering from LLM Internal States](https://mlsanity.com/paper/2607.11414/) (2026)
- [When the Wrong Key Wins: Understanding and Detecting Hallucinations in LLMs](https://mlsanity.com/paper/2609.15106/) (2026)

## Abstract (arXiv)

We investigate whether open-source LLMs encode a linearly separable truthfulness signal in their hidden states, and at which network depth this signal is strongest. Across three 7B--8B instruction-tuned models (Llama-3.1-8B, Mistral-7B, Qwen2.5-7B) loaded in 4-bit NF4 quantization, we extract per-layer hidden states on four hallucination benchmarks (TruthfulQA, HaluEval-QA, FEVER, and a controlled synthetic set) and compare four detection approaches: linear and MLP probes, INSIDE EigenScore, self-consistency, and attention entropy. A linear probe on a single mid-network layer achieves 0.904--1.000 AUROC on held-out splits, while sampling-based detectors do not exceed 0.541 AUROC under the same protocol. The truthfulness signal is approximately linear: MLP probes rarely surpass linear probes by more than 0.01 AUROC. Peak probing layers fall in a consistent band across model families on natural-language benchmarks -- blocks~13--18 of~32 for Llama and Mistral, and blocks~19--25 of~28 for Qwen. First-block attention entropy provides a complementary signal in knowledge-grounded settings (0.866--0.941 AUROC on HaluEval-QA) at no additional inference cost. The low discriminability of sampling methods under this protocol reflects a structural mismatch between paired-label evaluation and the information these methods access, rather than an inherent limitation of those methods. Code and data are released for full reproducibility on a single 8,GB GPU.