# The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems

arXiv: https://arxiv.org/abs/2503.03750 · HTML: https://mlsanity.com/paper/2503.03750/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 31 of its 39 references that are in the map.
- First posted on arXiv 5 Mar 2025.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 30 of 31)

- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://mlsanity.com/paper/2310.03214/) (2023)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [Frontier Models are Capable of In-context Scheming](https://mlsanity.com/paper/2412.04984/) (2024)
- [Do Large Language Models Know What They Don't Know?](https://mlsanity.com/paper/2305.18153/) (2023)
- [Inverse Scaling: When Bigger Isn't Better](https://mlsanity.com/paper/2306.09479/) (2023)
- [AI Deception: A Survey of Examples, Risks, and Potential Solutions](https://mlsanity.com/paper/2308.14752/) (2023)
- [Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark](https://mlsanity.com/paper/2304.03279/) (2023)
- [Explaining Answers with Entailment Trees](https://mlsanity.com/paper/2104.08661/) (2021)
- [Truthful AI: Developing and governing AI that does not lie](https://mlsanity.com/paper/2110.06674/) (2021)
- [CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge](https://mlsanity.com/paper/2109.01653/) (2021)
- [Alignment for Honesty](https://mlsanity.com/paper/2312.07000/) (2023)
- [Calibrating Large Language Models with Sample Consistency](https://mlsanity.com/paper/2402.13904/) (2024)
- [Prompt Consistency for Zero-Shot Task Generalization](https://mlsanity.com/paper/2205.00049/) (2022)
- [Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs](https://mlsanity.com/paper/2502.08640/) (2025)
- [Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?](https://mlsanity.com/paper/2407.21792/) (2024)
- [Do Large Language Models Know about Facts?](https://mlsanity.com/paper/2310.05177/) (2023)
- [e-CARE: a New Dataset for Exploring Explainable Causal Reasoning](https://mlsanity.com/paper/2205.05849/) (2022)
- [Evaluating Superhuman Models with Consistency Checks](https://mlsanity.com/paper/2306.09983/) (2023)
- [AI-LieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agents](https://mlsanity.com/paper/2409.09013/) (2024)
- [Localizing Lying in Llama: Understanding Instructed Dishonesty on True-False Questions Through Prompting, Probing, and Patching](https://mlsanity.com/paper/2311.15131/) (2023)
- [A Survey on the Honesty of Large Language Models](https://mlsanity.com/paper/2409.18786/) (2024)

## Built on by (most cited first, 30 of 72)

- [Thought Crime: Backdoors and Emergent Misalignment in Reasoning Models](https://mlsanity.com/paper/2506.13206/) (2025)
- [Detecting High-Stakes Interactions with Activation Probes](https://mlsanity.com/paper/2506.10805/) (2025)
- [Mitigating Deceptive Alignment via Self-Monitoring](https://mlsanity.com/paper/2505.18807/) (2025)
- [Training language models to be warm and empathetic makes them less reliable and more sycophantic](https://mlsanity.com/paper/2507.21919/) (2025)
- [Benchmarking Deception Probes via Black-to-White Performance Boosts](https://mlsanity.com/paper/2507.12691/) (2025)
- [Generative Value Conflicts Reveal LLM Priorities](https://mlsanity.com/paper/2509.25369/) (2025)
- [Difficulties with Evaluating a Deception Detector for AIs](https://mlsanity.com/paper/2511.22662/) (2025)
- [WOLF: Werewolf-based Observations for LLM Deception and Falsehoods](https://mlsanity.com/paper/2512.09187/) (2025)
- [The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probes](https://mlsanity.com/paper/2602.15515/) (2026)
- [Evaluating LLM Agent Collusion in Double Auctions](https://mlsanity.com/paper/2507.01413/) (2025)
- [The Singapore Consensus on Global AI Safety Research Priorities](https://mlsanity.com/paper/2506.20702/) (2025)
- [SafePro: Evaluating the Safety of Professional-Level AI Agents](https://mlsanity.com/paper/2601.06663/) (2026)
- [The Impact of Off-Policy Training Data on Probe Generalisation](https://mlsanity.com/paper/2511.17408/) (2025)
- [One Probe Won't Catch Them All: Towards Targeted Deception Detection](https://mlsanity.com/paper/2602.01425/) (2026)
- [Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures](https://mlsanity.com/paper/2609.29429/) (2026)
- [Can LLMs Lie? Investigation beyond Hallucination](https://mlsanity.com/paper/2509.03518/) (2025)
- [LH-Deception: Simulating and Understanding LLM Deceptive Behaviors in Long-Horizon Interactions](https://mlsanity.com/paper/2510.03999/) (2025)
- [Position: Anthropomorphic Misalignment Research Needs Stronger Evidence](https://mlsanity.com/paper/2606.07612/) (2026)
- [ManagerBench: Evaluating the Safety-Pragmatism Trade-off in Autonomous LLMs](https://mlsanity.com/paper/2510.00857/) (2025)
- [Beyond Prompt-Induced Lies: Investigating LLM Deception on Benign Prompts](https://mlsanity.com/paper/2508.06361/) (2025)
- [Automated Researchers Can Mitigate Well-characterized Alignment Failures](https://mlsanity.com/paper/2608.28945/) (2026)
- [Depth-Wise Activation Steering for Honest Language Models](https://mlsanity.com/paper/2512.07667/) (2025)
- [Activation Steering for Aligned Open-ended Generation without Sacrificing Coherence](https://mlsanity.com/paper/2604.08169/) (2026)
- [Decomposing Factual Sycophancy in Language Models: How Size and Instruction Tuning Shape Robustness](https://mlsanity.com/paper/2606.06306/) (2026)
- ["Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms](https://mlsanity.com/paper/2606.12618/) (2026)
- [Reinforcement Learning Towards Broadly and Persistently Beneficial Models](https://mlsanity.com/paper/2606.24014/) (2026)
- [Constitutional Midtraining: Content Presence Drives Alignment Gains](https://mlsanity.com/paper/2607.26654/) (2026)
- [Quantifying Overclaiming Propensity in Frontier LLM Agents](https://mlsanity.com/paper/2609.20812/) (2026)
- [But what is your honest answer? Aiding LLM-judges with honest alternatives using steering vectors](https://mlsanity.com/paper/2505.17760/) (2025)
- [AI Must not be Fully Autonomous](https://mlsanity.com/paper/2507.23330/) (2025)

## Abstract (arXiv)

As large language models (LLMs) become more capable and agentic, the requirement for trust in their outputs grows significantly, yet at the same time concerns have been mounting that models may learn to lie in pursuit of their goals. To address these concerns, a body of work has emerged around the notion of "honesty" in LLMs, along with interventions aimed at mitigating deceptive behaviors. However, some benchmarks claiming to measure honesty in fact simply measure accuracy--the correctness of a model's beliefs--in disguise. Moreover, no benchmarks currently exist for directly measuring whether language models lie. In this work, we introduce a large-scale human-collected dataset for directly measuring lying, allowing us to disentangle accuracy from honesty. Across a diverse set of LLMs, we find that while larger models obtain higher accuracy on our benchmark, they do not become more honest. Surprisingly, most frontier LLMs obtain high scores on truthfulness benchmarks yet exhibit a substantial propensity to lie under pressure, resulting in low honesty scores on our benchmark. We find that simple methods, such as representation engineering interventions, can improve honesty. These results underscore the growing need for robust evaluations and effective interventions to ensure LLMs remain trustworthy.