mlsanityData to 7 Oct 2026

How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions

arXiv 2309.15840 ↗First posted 26 Sep 2023cs.CL cs.AI cs.LGcited by 82

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) can "lie", which we define as outputting false statements despite "knowing" the truth in a demonstrable sense. LLMs might "lie", for example, when instructed to output misinformation. Here, we develop a simple lie detector that requires neither access to the LLM's activations (black-box) nor ground-truth knowledge of the fact in question. The detector works by asking a predefined set of unrelated follow-up questions after a suspected lie, and feeding the LLM's yes/no answers into a logistic regression classifier. Despite its simplicity, this lie detector is highly accurate and surprisingly general. When trained on examples from a single setting -- prompting GPT-3.5 to lie about factual questions -- the detector generalises out-of-distribution to (1) other LLM architectures, (2) LLMs fine-tuned to lie, (3) sycophantic lies, and (4) lies emerging in real-life scenarios such as sales. These results indicate that LLMs have distinctive lie-related behavioural patterns, consistent across architectures and contexts, which could enable general-purpose lie detection.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Builds on 31 of its 57 references that are in the map. First posted on arXiv 26 Sep 2023.

Builds on

31 of 57 references.

References · 31

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  7. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  8. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  9. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  10. 2021Ethical and social risks of harm from Language Modelscited by 957
  11. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  12. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
19 more
  1. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  4. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  5. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  6. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  7. 2022Self-critiquing models for assisting human evaluatorscited by 372
  8. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  9. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  10. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
  11. 2023Simple synthetic data reduces sycophancy in large language modelscited by 174
  12. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 155
  13. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 123
  14. 2021Truthful AI: Developing and governing AI that does not liecited by 109
  15. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 108
  16. 2023Evaluating Superhuman Models with Consistency Checkscited by 47
  17. 2023Hoodwinked: Deception and Cooperation in a Text-Based Game for Language Modelscited by 46
  18. 2023Characterizing Manipulation from AI Systemscited by 43
  19. 2023Artificial Influence: An Analysis Of AI-Driven Persuasioncited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 27; 2025: 25; 2026: 22.

82 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  3. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  4. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  5. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 183
  6. 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 132
  7. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 131
  8. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  9. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 111
  10. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 105
  11. 2024Truth is Universal: Robust Detection of Lies in LLMscited by 78
  12. 2023Alignment for Honestycited by 72
38 more
  1. 2024Calibrating Large Language Models Using Their Generations Onlycited by 63
  2. 2024Factual Confidence of LLMs: on Reliability and Robustness of Current Estimatorscited by 59
  3. 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 57
  4. 2023Honesty Is the Best Policy: Defining and Mitigating AI Deceptioncited by 40
  5. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 36
  6. 2025Probing and Steering Evaluation Awareness of Language Modelscited by 31
  7. 2023Challenges with unsupervised LLM knowledge discoverycited by 30
  8. 2024AI-LieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agentscited by 28
  9. 2025Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answercited by 26
  10. 2023Localizing Lying in Llama: Understanding Instructed Dishonesty on True-False Questions Through Prompting, Probing, and Patchingcited by 24
  11. 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 24
  12. 2024The Reasons that Agents Act: Intention and Instrumental Goalscited by 23
  13. 2024Noise Injection Reveals Hidden Capabilities of Sandbagging Language Modelscited by 22
  14. 2024BeHonest: Benchmarking Honesty in Large Language Modelscited by 20
  15. 2024Constructing Benchmarks and Interventions for Combating Hallucinations in LLMscited by 19
  16. 2025Robust Hallucination Detection in LLMs via Adaptive Token Selectioncited by 19
  17. 2025Probing the Geometry of Truth: Consistency and Generalization of Truth Directions in LLMs Across Logical Transformations and Question Answering Taskscited by 18
  18. 2024Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Artcited by 17
  19. 2025Benchmarking Deception Probes via Black-to-White Performance Boostscited by 17
  20. 2024Just rephrase it! Uncertainty estimation in closed-source language models via multiple rephrased queriescited by 15
  21. 2024Distinguishing Ignorance from Error in LLM Hallucinationscited by 14
  22. 2024Lies, Damned Lies, and Distributional Language Statistics: Persuasion and Deception with Large Language Modelscited by 14
  23. 2024Poser: Unmasking Alignment Faking LLMs by Manipulating Their Internalscited by 13
  24. 2024LLMScan: Causal Scan for LLM Misbehavior Detectioncited by 13
  25. 2024Evaluating AI Evaluation: Perils and Prospectscited by 11
  26. 2025Do Large Language Models Know How Much They Know?cited by 11
  27. 2025Difficulties with Evaluating a Deception Detector for AIscited by 11
  28. 2025Predicting the Performance of Black-box LLMs through Follow-up Queriescited by 9
  29. 2025When Truthful Representations Flip Under Deceptive Instructions?cited by 9
  30. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  31. 2024Efficient Model-agnostic Alignment via Bayesian Persuasioncited by 8
  32. 2025Preference Learning with Lie Detectors can Induce Honesty or Evasioncited by 8
  33. 2024Transferable and Efficient Non-Factual Content Detection via Probe Training with Offline Consistency Checkingcited by 7
  34. 2025Evaluating & Reducing Deceptive Dialogue From Language Models with Multi-turn RLcited by 7
  35. 2026LLMs Know They're Wrong and Agree Anyway: The Shared Sycophancy-Lying Circuitcited by 7
  36. 2025Mechanistic Interpretability Needs Philosophycited by 6
  37. 2025HACK: Hallucinations Along Certainty and Knowledge Axescited by 6
  38. 2024Meta-Models: An Architecture for Decoding LLM Behaviors Through Interpreted Embeddings and Natural Languagecited by 5

50 most cited of 82 All 82 (JSON)