mlsanityData to 14 Oct 2026

Stress-Testing LLM Lie Detectors: Role-Play Failures and Spurious Correlations

arXiv 2609.39807 ↗First posted 30 Sep 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Lie detection probes aim to predict from a language model's internal states whether its output is truthful or dishonest. However, role-play complicates what "truth" means for an LLM: language models can adopt a wide range of personas that take very different claims to be true, including personas whose beliefs clearly contradict reality, such as a conspiracy theorist. In this work, we investigate whether lie detection probes reliably flag falsehoods generated under such an anti-factual persona or whether they instead follow the persona's beliefs. We introduce a dataset of 8,916 human-reviewed, on-policy responses from three LLMs adopting anti-factual personas. Evaluating eight probes from prior work, we find that many fail in this setting, particularly when correct and incorrect answers are evaluated under the same persona prompt. To investigate why, we construct three novel confounder datasets in which truth is anti-correlated with a potential confounding concept. Our experiments reveal that many existing probes strongly track concepts that are spuriously correlated with truth in their training data, such as instruction compliance or response likelihood. Based on these findings, we introduce a simple linear probe that achieves the strongest overall performance on both the persona and confounder stress tests. Our results suggest that current lie detection probes are far from reliable and highlight the need for training data in which truth is decorrelated from confounding concepts.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 39 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

31 of 39 references.

References · 31

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  2. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  3. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  5. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  6. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  7. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
  8. 2024Alignment faking in large language modelscited by 360
  9. 2023Role-Play with Large Language Modelscited by 294
  10. 2024Frontier Models are Capable of In-context Schemingcited by 287
  11. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 188
  12. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 145
19 more
  1. 2022Language Models as Agent Modelscited by 132
  2. 2023Deception Abilities Emerged in Large Language Modelscited by 101
  3. 2023Scheming AIs: Will AIs fake alignment during training in order to get power?cited by 97
  4. 2026The Assistant Axis: Situating and Stabilizing the Default Persona of Language Modelscited by 90
  5. 2023How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questionscited by 83
  6. 2024Truth is Universal: Robust Detection of Lies in LLMscited by 79
  7. 2023Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblockscited by 78
  8. 2024Towards evaluations-based safety cases for AI schemingcited by 33
  9. 2024Standards for Belief Representations in LLMscited by 30
  10. 2024Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?cited by 24
  11. 2025Believe It or Not: How Deeply do LLMs Believe Implanted Facts?cited by 24
  12. 2025Liars' Bench: Evaluating Lie Detectors for Language Modelscited by 22
  13. 2025Benchmarking Deception Probes via Black-to-White Performance Boostscited by 19
  14. 2025Difficulties with Evaluating a Deception Detector for AIscited by 13
  15. 2026Linear representations in language models can change dramatically over a conversationcited by 10
  16. 2026One Probe Won't Catch Them All: Towards Targeted Deception Detectioncited by 9
  17. 2024Truth-value judgment in language models: 'truth directions' are context sensitivecited by 7
  18. 2026When Role-playing, Do Models Believe What They Say?cited by 4
  19. 2026"Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organismscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Can LLMs Lie? Investigation beyond Hallucination
  2. 2026Beyond Liars' Bench: The Impact of Lie Typology, Depth, and Sparsity on Deception Detection in LLMs
  3. 2025Reasoning Isn't Enough: Examining Truth-Bias and Sycophancy in LLMs
  4. 2025Are the Hidden States Hiding Something? Testing the Limits of Factuality-Encoding Capabilities in LLMs
  5. 2025Beyond Prompt-Induced Lies: Investigating LLM Deception on Benign Prompts
  6. 2025Evaluating & Reducing Deceptive Dialogue From Language Models with Multi-turn RL
  7. 2026Censored LLMs as a Natural Testbed for Secret Knowledge Elicitation
  8. 2026Disentangling Deception and Hallucination Failures in LLMs