# Do LLMs Know about Hallucination? An Empirical Investigation of LLM's Hidden States

arXiv: https://arxiv.org/abs/2402.09733 · HTML: https://mlsanity.com/paper/2402.09733/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 35 references that are in the map.
- First posted on arXiv 15 Feb 2024.

## Builds on (most cited first, 18 of 18)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Trusting Your Evidence: Hallucinate Less with Context-aware Decoding](https://mlsanity.com/paper/2305.14739/) (2023)
- [FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios](https://mlsanity.com/paper/2307.13528/) (2023)
- [Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies](https://mlsanity.com/paper/2308.03188/) (2023)
- [Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning](https://mlsanity.com/paper/2305.14160/) (2023)
- [The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediations](https://mlsanity.com/paper/2310.04988/) (2023)
- [Complex Claim Verification with Evidence Retrieved in the Wild](https://mlsanity.com/paper/2305.11859/) (2023)
- [Jump to Conclusions: Short-Cutting Transformers With Linear Transformations](https://mlsanity.com/paper/2303.09435/) (2023)
- [Controlled Hallucinations: Learning to Generate Faithfully from Noisy Data](https://mlsanity.com/paper/2010.05873/) (2020)

## Built on by (most cited first, 30 of 52)

- [HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detection](https://mlsanity.com/paper/2409.17504/) (2024)
- [Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approach](https://mlsanity.com/paper/2404.15993/) (2024)
- [Large Language Model Safety: A Holistic Survey](https://mlsanity.com/paper/2412.17686/) (2024)
- [Efficiently Scaling LLM Reasoning with Certaindex](https://mlsanity.com/paper/2412.20993/) (2024)
- [Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluation](https://mlsanity.com/paper/2410.13640/) (2024)
- [Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Era](https://mlsanity.com/paper/2403.08946/) (2024)
- [Safety-Aware Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2410.10014/) (2024)
- [Do LLMs "know" internally when they follow instructions?](https://mlsanity.com/paper/2410.14516/) (2024)
- [Reference-free Hallucination Detection for Large Vision-Language Models](https://mlsanity.com/paper/2408.05767/) (2024)
- [Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputs](https://mlsanity.com/paper/2510.01218/) (2025)
- [DebUnc: Improving Large Language Model Agent Communication With Uncertainty Metrics](https://mlsanity.com/paper/2407.06426/) (2024)
- [Can LLMs Predict Their Own Failures? Self-Awareness via Internal Circuits](https://mlsanity.com/paper/2512.20578/) (2025)
- [When Context Leads but Parametric Memory Follows in Large Language Models](https://mlsanity.com/paper/2409.08435/) (2024)
- [FactCheckmate: Preemptively Detecting and Mitigating Hallucinations in LMs](https://mlsanity.com/paper/2410.02899/) (2024)
- [LUMIA: Linear probing for Unimodal and MultiModal Membership Inference Attacks leveraging internal LLM states](https://mlsanity.com/paper/2411.19876/) (2024)
- [Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety](https://mlsanity.com/paper/2506.05451/) (2025)
- [Quantifying and Mitigating Self-Preference Bias of LLM Judges](https://mlsanity.com/paper/2604.22891/) (2026)
- [Shadows in the Attention: Contextual Perturbation and Representation Drift in the Dynamics of Hallucination in LLMs](https://mlsanity.com/paper/2505.16894/) (2025)
- [Bridging Visualization and Optimization: Multimodal Large Language Models on Graph-Structured Combinatorial Optimization](https://mlsanity.com/paper/2501.11968/) (2025)
- [Explainable XR: Understanding User Behaviors of XR Environments using LLM-assisted Analytics Framework](https://mlsanity.com/paper/2501.13778/) (2025)
- [Exposing the Ghost in the Transformer: Abnormal Detection for Large Language Models via Hidden State Forensics](https://mlsanity.com/paper/2504.00446/) (2025)
- [JAM: Controllable and Responsible Text Generation via Causal Reasoning and Latent Vector Manipulation](https://mlsanity.com/paper/2502.20684/) (2025)
- [Do Large Language Models (Really) Need Statistical Foundations?](https://mlsanity.com/paper/2505.19145/) (2025)
- [CoRE: Enhancing Metacognition with Label-free Self-evaluation in LRMs](https://mlsanity.com/paper/2507.06087/) (2025)
- [ReasoningBomb: A Stealthy Denial-of-Service Attack by Inducing Pathologically Long Reasoning in Large Reasoning Models](https://mlsanity.com/paper/2602.00154/) (2026)
- [Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers?](https://mlsanity.com/paper/2404.07066/) (2024)
- [Monitoring Decoding: Mitigating Hallucination via Evaluating the Factuality of Partial Response during Generation](https://mlsanity.com/paper/2503.03106/) (2025)
- [Latte: Transfering LLMs` Latent-level Knowledge for Few-shot Tabular Learning](https://mlsanity.com/paper/2505.05237/) (2025)
- [LPASS: Linear Probes as Stepping Stones for vulnerability detection using compressed LLMs](https://mlsanity.com/paper/2505.24451/) (2025)
- [Language Surgery in Multilingual Large Language Models](https://mlsanity.com/paper/2506.12450/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) can make up answers that are not real, and this is known as hallucination. This research aims to see if, how, and to what extent LLMs are aware of hallucination. More specifically, we check whether and how an LLM reacts differently in its hidden states when it answers a question right versus when it hallucinates. To do this, we introduce an experimental framework which allows examining LLM's hidden states in different hallucination situations. Building upon this framework, we conduct a series of experiments with language models in the LLaMA family (Touvron et al., 2023). Our empirical findings suggest that LLMs react differently when processing a genuine response versus a fabricated one. We then apply various model interpretation techniques to help understand and explain the findings better. Moreover, informed by the empirical observations, we show great potential of using the guidance derived from LLM's hidden representation space to mitigate hallucination. We believe this work provides insights into how LLMs produce hallucinated answers and how to make them occur less often.