mlsanityData to 14 Oct 2026

Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?

arXiv 2312.03729 ↗First posted 27 Nov 2023cs.CL cs.AIcited by 62

Not on the map. Why?

Abstract (arXiv)

Neural language models (LMs) can be used to evaluate the truth of factual statements in two ways: they can be either queried for statement probabilities, or probed for internal representations of truthfulness. Past work has found that these two procedures sometimes disagree, and that probes tend to be more accurate than LM outputs. This has led some researchers to conclude that LMs "lie" or otherwise encode non-cooperative communicative intents. Is this an accurate description of today's LMs, or can query-probe disagreement arise in other ways? We identify three different classes of disagreement, which we term confabulation, deception, and heterogeneity. In many cases, the superiority of probes is simply attributable to better calibration on uncertain answers rather than a greater fraction of correct, high-confidence answers. In some cases, queries and probes perform better on different subsets of inputs, and accuracy can further be improved by ensembling the two. Code is available at github.com/lingo-mit/lm-truthfulness.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Builds on 15 of its 26 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

15 of 26 references.

References · 15

  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  3. 2019Language Models as Knowledge Bases?cited by 1,936
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  5. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  7. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  8. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  9. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  10. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 232
  11. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  12. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 151
3 more
  1. 2021Truthful AI: Developing and governing AI that does not liecited by 113
  2. 2021CREAK: A Dataset for Commonsense Reasoning over Entity Knowledgecited by 88
  3. 2021The Low-Dimensional Linear Geometry of Contextualized Word Representationscited by 40

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 26; 2025: 15; 2026: 19.

62 papers.

Sort:

By area:

  1. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 244
  2. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 163
  3. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  4. 2024Unfamiliar Finetuning Examples Control How Language Models Hallucinatecited by 87
  5. 2024LLM Internal States Reveal Hallucination Risk Faced With a Querycited by 82
  6. 2024Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approachcited by 76
  7. 2023Do Androids Know They're Only Dreaming of Electric Sheep?cited by 67
  8. 2024Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluationcited by 57
  9. 2025Detecting High-Stakes Interactions with Activation Probescited by 53
  10. 2024Confabulation: The Surprising Value of Large Language Model Hallucinationscited by 43
  11. 2024Insights into LLM Long-Context Failures: When Transformers Know but Don't Tellcited by 36
  12. 2026TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Roboticscited by 33
38 more
  1. 2024Understanding Finetuning for Factual Knowledge Extractioncited by 32
  2. 2024Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluationcited by 29
  3. 2024FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluationcited by 25
  4. 2024A Survey on the Honesty of Large Language Modelscited by 23
  5. 2023A Study on the Calibration of In-context Learningcited by 22
  6. 2024Language Models in Dialogue: Conversational Maxims for Human-AI Interactionscited by 21
  7. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 21
  8. 2025Probing the Geometry of Truth: Consistency and Generalization of Truth Directions in LLMs Across Logical Transformations and Question Answering Taskscited by 18
  9. 2024Rel-A.I.: An Interaction-Centered Approach To Measuring Human-LM Reliancecited by 16
  10. 2025Investigating task-specific prompts and sparse autoencoders for activation monitoringcited by 15
  11. 2025Enhancing Hallucination Detection through Noise Injectioncited by 13
  12. 2024Exploring Multilingual Probing in Large Language Models: A Cross-Language Analysiscited by 12
  13. 2025Emergence of Linear Truth Encodings in Language Modelscited by 12
  14. 2026Can LLMs Introspect? A Reality Checkcited by 12
  15. 2024LLMs Faithfully and Iteratively Compute Answers During CoT: A Systematic Analysis With Multi-step Arithmeticscited by 11
  16. 2024Prospects for inconsistency detection using large language models and sheavescited by 10
  17. 2024Brittle Minds, Fixable Activations: Understanding Belief Representations in Language Modelscited by 8
  18. 2026Interpretability Can Be Actionablecited by 7
  19. 2024Online Safety Analysis for LLMs: a Benchmark, an Assessment, and a Path Forwardcited by 6
  20. 2025Bridging the Knowledge-Prediction Gap in LLMs on Multiple-Choice Questionscited by 6
  21. 2026Reaching Beyond the Mode: RL for Distributional Reasoning in Language Modelscited by 6
  22. 2025REMA: A Unified Reasoning Manifold Framework for Interpreting Large Language Modelcited by 5
  23. 2026Hallucinations Undermine Trust; Metacognition is a Way Forwardcited by 5
  24. 2024A Bayesian Approach to Harnessing the Power of LLMs in Authorship Attributioncited by 4
  25. 2025The Curious Case of Factuality Finetuning: Models' Internal Beliefs Can Improve Factualitycited by 4
  26. 2025Humans overrely on overconfident language models, across languagescited by 3
  27. 2025Efficient Knowledge Probing of Large Language Models by Adapting Pre-trained Embeddingscited by 3
  28. 2026Indications of Belief-Guided Agency and Meta-Cognitive Monitoring in Large Language Modelscited by 3
  29. 2025ISACL: Internal State Analyzer for Copyrighted Training Data Leakagecited by 2
  30. 2026Testing the Limits of Truth Directions in LLMscited by 2
  31. 2024Uncovering Latent Human Wellbeing in Language Model Embeddingscited by 1
  32. 2024What Would You Ask When You First Saw a2+b2=c2? Evaluating LLM on Curiosity-Driven Questioningcited by 1
  33. 2024Listening to the Wise Few: Query-Key Alignment Unlocks Latent Correct Answers in Large Language Modelscited by 1
  34. 2025Continuum-Interaction-Driven Intelligence: Human-Aligned Neural Architecture via Crystallized Reasoning and Fluid Generationcited by 1
  35. 2025Express Your Doubts -- Probabilistic World Modeling Should not be Based on Token logprobscited by 1
  36. 2025Exploring the generalization of LLM truth directions on conversational formatscited by 1
  37. 2026Nodes Are Early, Edges Are Late: Probing Diagram Representations in Large Vision-Language Modelscited by 1
  38. 2026Diagnosing Correctness Probes under Self-Judgement Confoundingcited by 1

50 most cited of 62 All 62 (JSON)