mlsanityData to 14 Oct 2026

The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems

arXiv 2503.03750 ↗First posted 5 Mar 2025cs.LG cs.AI cs.CLcited by 72

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

As large language models (LLMs) become more capable and agentic, the requirement for trust in their outputs grows significantly, yet at the same time concerns have been mounting that models may learn to lie in pursuit of their goals. To address these concerns, a body of work has emerged around the notion of "honesty" in LLMs, along with interventions aimed at mitigating deceptive behaviors. However, some benchmarks claiming to measure honesty in fact simply measure accuracy--the correctness of a model's beliefs--in disguise. Moreover, no benchmarks currently exist for directly measuring whether language models lie. In this work, we introduce a large-scale human-collected dataset for directly measuring lying, allowing us to disentangle accuracy from honesty. Across a diverse set of LLMs, we find that while larger models obtain higher accuracy on our benchmark, they do not become more honest. Surprisingly, most frontier LLMs obtain high scores on truthfulness benchmarks yet exhibit a substantial propensity to lie under pressure, resulting in low honesty scores on our benchmark. We find that simple methods, such as representation engineering interventions, can improve honesty. These results underscore the growing need for robust evaluations and effective interventions to ensure LLMs remain trustworthy.

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 31 of its 39 references that are in the map. First posted on arXiv 5 Mar 2025.

Builds on

31 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024AI-LieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agents13 Sep

Other references · 30

  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  4. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  5. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  6. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  7. 2024Alignment faking in large language modelscited by 360
  8. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  9. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  10. 2023Language Models Represent Space and Timecited by 316
  11. 2021Unsolved Problems in ML Safetycited by 292
  12. 2024Frontier Models are Capable of In-context Schemingcited by 287
18 more
  1. 2023Do Large Language Models Know What They Don't Know?cited by 243
  2. 2023Inverse Scaling: When Bigger Isn't Bettercited by 207
  3. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 188
  4. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 182
  5. 2021Explaining Answers with Entailment Treescited by 152
  6. 2021Truthful AI: Developing and governing AI that does not liecited by 113
  7. 2021CREAK: A Dataset for Commonsense Reasoning over Entity Knowledgecited by 88
  8. 2023Alignment for Honestycited by 74
  9. 2024Calibrating Large Language Models with Sample Consistencycited by 73
  10. 2022Prompt Consistency for Zero-Shot Task Generalizationcited by 67
  11. 2025Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIscited by 67
  12. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 61
  13. 2023Do Large Language Models Know about Facts?cited by 51
  14. 2022e-CARE: a New Dataset for Exploring Explainable Causal Reasoningcited by 47
  15. 2023Evaluating Superhuman Models with Consistency Checkscited by 47
  16. 2023Localizing Lying in Llama: Understanding Instructed Dishonesty on True-False Questions Through Prompting, Probing, and Patchingcited by 24
  17. 2024A Survey on the Honesty of Large Language Modelscited by 23
  18. 2024Does ChatGPT Have a Mind?cited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 19; 2026: 53.

72 papers.

Sort:

By area:

  1. 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 65
  2. 2025Detecting High-Stakes Interactions with Activation Probescited by 53
  3. 2025Mitigating Deceptive Alignment via Self-Monitoringcited by 29
  4. 2025Training language models to be warm and empathetic makes them less reliable and more sycophanticcited by 21
  5. 2025Benchmarking Deception Probes via Black-to-White Performance Boostscited by 19
  6. 2025Generative Value Conflicts Reveal LLM Prioritiescited by 13
  7. 2025Difficulties with Evaluating a Deception Detector for AIscited by 13
  8. 2025WOLF: Werewolf-based Observations for LLM Deception and Falsehoodscited by 13
  9. 2026The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probescited by 12
  10. 2025Evaluating LLM Agent Collusion in Double Auctionscited by 11
  11. 2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 10
  12. 2026SafePro: Evaluating the Safety of Professional-Level AI Agentscited by 10
38 more
  1. 2025The Impact of Off-Policy Training Data on Probe Generalisationcited by 9
  2. 2026One Probe Won't Catch Them All: Towards Targeted Deception Detectioncited by 9
  3. 2026Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failurescited by 8
  4. 2025Can LLMs Lie? Investigation beyond Hallucinationcited by 7
  5. 2025LH-Deception: Simulating and Understanding LLM Deceptive Behaviors in Long-Horizon Interactionscited by 7
  6. 2026Position: Anthropomorphic Misalignment Research Needs Stronger Evidencecited by 7
  7. 2025ManagerBench: Evaluating the Safety-Pragmatism Trade-off in Autonomous LLMscited by 6
  8. 2025Beyond Prompt-Induced Lies: Investigating LLM Deception on Benign Promptscited by 5
  9. 2026Automated Researchers Can Mitigate Well-characterized Alignment Failurescited by 5
  10. 2025Depth-Wise Activation Steering for Honest Language Modelscited by 4
  11. 2026Activation Steering for Aligned Open-ended Generation without Sacrificing Coherencecited by 4
  12. 2026Decomposing Factual Sycophancy in Language Models: How Size and Instruction Tuning Shape Robustnesscited by 4
  13. 2026"Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organismscited by 4
  14. 2026Reinforcement Learning Towards Broadly and Persistently Beneficial Modelscited by 4
  15. 2026Constitutional Midtraining: Content Presence Drives Alignment Gainscited by 4
  16. 2026Quantifying Overclaiming Propensity in Frontier LLM Agentscited by 4
  17. 2025But what is your honest answer? Aiding LLM-judges with honest alternatives using steering vectorscited by 3
  18. 2025AI Must not be Fully Autonomouscited by 3
  19. 2026ForesightSafety Bench: A Frontier Risk Evaluation and Governance Framework towards Safe AIcited by 3
  20. 2026Boiling the Frog: A Multi-Turn Benchmark for Agentic Safetycited by 3
  21. 2026Position: Correct Answer, Wrong Mechanism -- When AI Scientists Defend General Claims Their Own Data Contradictscited by 3
  22. 2026Chasing the Public Score: User Pressure and Evaluation Exploitation in Coding Agent Workflowscited by 2
  23. 2026LLMs Contain Multitudes: How Deployment Context Reshapes Model-Level Preferences and Valuescited by 2
  24. 2026Decoding Hidden Deception in Reasoning LLMs: Activation Explainers for Deception Auditingcited by 2
  25. 2026Knowledge-Verified Emergent Deception in LLM Agents Under Conflicting Incentivescited by 2
  26. 2026DialDefer: A Framework for Detecting and Mitigating LLM Dialogic Deferencecited by 1
  27. 2026Pressure Reveals Character: Behavioural Alignment Evaluation at Depthcited by 1
  28. 2026Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearningcited by 1
  29. 2026Muse Spark Safety & Preparedness Reportcited by 1
  30. 2026RogueAI: A Reverse Turing Test for Detecting Licensed AI Deception in Dialoguecited by 1
  31. 2026Probing the Misaligned Thinking Process of Language Modelscited by 1
  32. 2026Scaling Trends for Lie Detector Oversight in Preference Learningcited by 1
  33. 2026Evaluation Awareness in Language Models: Representation, Verbalization, and Controlcited by 1
  34. 2026Beliefs and Behavior in Language Modelscited by 1
  35. 2025AI Identity Disclosure Under Professional Personas: A Gap Between Capacity and Consistencycited by 0
  36. 2026Stable Reasoning, Unstable Responses: Mitigating LLM Deception via Stability Asymmetrycited by 0
  37. 2026Code World Model Preparedness Reportcited by 0
  38. 2026Triadic Werewolf: A Jester Role for Multi-Hop Theory of Mind in LLMscited by 0

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025When Thinking LLMs Lie: Unveiling the Strategic Deception in Representations of Reasoning Models
  2. 2026Truthful AI Advisors: A Pre-Specified Benchmark for Large Language Model Honesty Under Preference Misalignment
  3. 2026Lying to Win: Assessing LLM Deception through Human-AI Games and Parallel-World Probing
  4. 2026Think Before You Lie: How Reasoning Leads to Honesty
  5. 2026Consensus is Not Verification: Why Crowd Wisdom Strategies Fail for LLM Truthfulness
  6. 2025Among Us: A Sandbox for Measuring and Detecting Agentic Deception
  7. 2025DeceptionBench: A Comprehensive Benchmark for AI Deception Behaviors in Real-world Scenarios
  8. 2026Censored LLMs as a Natural Testbed for Secret Knowledge Elicitation