# "Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms

arXiv: https://arxiv.org/abs/2606.12618 · HTML: https://mlsanity.com/paper/2606.12618/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 10 Jun 2026.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 21 of 21)

- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [AI Control: Improving Safety Despite Intentional Subversion](https://mlsanity.com/paper/2312.06942/) (2023)
- [How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions](https://mlsanity.com/paper/2309.15840/) (2023)
- [The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems](https://mlsanity.com/paper/2503.03750/) (2025)
- [Detecting Strategic Deception Using Linear Probes](https://mlsanity.com/paper/2502.03407/) (2025)
- [Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers](https://mlsanity.com/paper/2512.15674/) (2025)
- [Safety Cases: How to Justify the Safety of Advanced AI Systems](https://mlsanity.com/paper/2403.10462/) (2024)
- [Stress-Testing Capability Elicitation With Password-Locked Models](https://mlsanity.com/paper/2405.19550/) (2024)
- [A sketch of an AI control safety case](https://mlsanity.com/paper/2501.17315/) (2025)
- [AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors](https://mlsanity.com/paper/2602.22755/) (2026)
- [Eliciting Secret Knowledge from Language Models](https://mlsanity.com/paper/2510.01070/) (2025)
- [Reasoning Models Struggle to Control their Chains of Thought](https://mlsanity.com/paper/2603.05706/) (2026)
- [Liars' Bench: Evaluating Lie Detectors for Language Models](https://mlsanity.com/paper/2511.16035/) (2025)
- [Towards eliciting latent knowledge from LLMs with mechanistic interpretability](https://mlsanity.com/paper/2505.14352/) (2025)
- [Benchmarking Deception Probes via Black-to-White Performance Boosts](https://mlsanity.com/paper/2507.12691/) (2025)
- [Difficulties with Evaluating a Deception Detector for AIs](https://mlsanity.com/paper/2511.22662/) (2025)
- [Introspection Adapters: Training LLMs to Report Their Learned Behaviors](https://mlsanity.com/paper/2604.16812/) (2026)
- [One Probe Won't Catch Them All: Towards Targeted Deception Detection](https://mlsanity.com/paper/2602.01425/) (2026)

## Built on by (most cited first, 4 of 4)

- [Stress-Testing LLM Lie Detectors: Role-Play Failures and Spurious Correlations](https://mlsanity.com/paper/2609.39807/) (2026)
- [Pre-training interventions, ex post facto: Grafting model beliefs across checkpoints](https://mlsanity.com/paper/2610.00767/) (2026)
- [Caught in the Act: Probes Effectively Detect Sabotage and Catch Unverbalized Deception](https://mlsanity.com/paper/2610.12445/) (2026)
- [On the Monitorability of Natural Language Autoencoders: Identification vs Discrimination](https://mlsanity.com/paper/2610.18727/) (2026)

## Abstract (arXiv)

Robust lie detectors for language models could enable powerful techniques for auditing, monitoring, and post-hoc investigation of model behaviour, but evaluating them requires testbeds where models verifiably believe the opposite of what they say. We show that existing trained model organisms often fail this requirement, leaving prior positive and negative detection results difficult to interpret. We address this with 13 reasoning model organisms whose hidden beliefs are verified in chain-of-thought and shown to generalise to held-out tasks, alongside Varied Deception, a prompted-lying testbed covering a broad range of lie-inducing motivations. On these testbeds we evaluate four detectors: a chain-of-thought judge, a logprob classifier, and two activation probes, including Did-You-Lie (DYL), a new method for training follow-up probes. On prompted lying, across 31 open-weight models spanning 2B to 1T parameters, all four detectors show positive scaling with model capability. However, every activation- and logprob-based detector drops sharply on our trained model organisms, with DYL retaining the most signal; only the chain-of-thought judge remains strong, with 0.82 balanced accuracy, partly as an artefact of our verification process favouring CoT-readable beliefs. Current lie detectors therefore cannot support high-confidence claims about model beliefs, and we suggest research directions that may address some of their current limitations. We release our datasets, model organisms, and trained detectors.