# BeHonest: Benchmarking Honesty in Large Language Models

arXiv: https://arxiv.org/abs/2406.13261 · HTML: https://mlsanity.com/paper/2406.13261/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 70 references that are in the map.
- First posted on arXiv 19 Jun 2024.

## Builds on (most cited first, 30 of 64)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)

## Built on by (most cited first, 24 of 24)

- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [A Survey on the Honesty of Large Language Models](https://mlsanity.com/paper/2409.18786/) (2024)
- [DeceptionBench: A Comprehensive Benchmark for AI Deception Behaviors in Real-world Scenarios](https://mlsanity.com/paper/2510.15501/) (2025)
- [Risk Analysis Techniques for Governed LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2508.05687/) (2025)
- [Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges](https://mlsanity.com/paper/2507.19672/) (2025)
- [The Secret Agenda: LLMs Strategically Lie and Our Current Safety Tools Are Blind](https://mlsanity.com/paper/2509.20393/) (2025)
- [Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions](https://mlsanity.com/paper/2501.11790/) (2025)
- [D-REX: A Benchmark for Detecting Deceptive Reasoning in Large Language Models](https://mlsanity.com/paper/2509.17938/) (2025)
- [Reasoning Models Will Sometimes Lie About Their Reasoning](https://mlsanity.com/paper/2601.07663/) (2026)
- [Evaluating Language Models for Harmful Manipulation](https://mlsanity.com/paper/2603.25326/) (2026)
- [Multilevel Interpretability Of Artificial Neural Networks: Leveraging Framework And Methods From Neuroscience](https://mlsanity.com/paper/2408.12664/) (2024)
- [MTCMB: A Multi-Task Benchmark Framework for Evaluating LLMs on Knowledge, Reasoning, and Safety in Traditional Chinese Medicine](https://mlsanity.com/paper/2506.01252/) (2025)
- [PRISON: Unmasking the Criminal Potential of Large Language Models](https://mlsanity.com/paper/2506.16150/) (2025)
- [Knowledge-Verified Emergent Deception in LLM Agents Under Conflicting Incentives](https://mlsanity.com/paper/2608.26372/) (2026)
- [I'm Sorry Dave: How the old world of personnel security can inform the new world of AI insider risk](https://mlsanity.com/paper/2504.00012/) (2025)
- [RogueAI: A Reverse Turing Test for Detecting Licensed AI Deception in Dialogue](https://mlsanity.com/paper/2606.13310/) (2026)
- [AI Identity Disclosure Under Professional Personas: A Gap Between Capacity and Consistency](https://mlsanity.com/paper/2511.21569/) (2025)
- [Responsible AI for General-Purpose Systems: Overview, Challenges, and A Path Forward](https://mlsanity.com/paper/2601.13122/) (2026)
- [Stable Reasoning, Unstable Responses: Mitigating LLM Deception via Stability Asymmetry](https://mlsanity.com/paper/2603.26846/) (2026)
- [DECOR: Auditing LLM Deception via Information Manipulation Theory](https://mlsanity.com/paper/2605.19270/) (2026)
- [From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch](https://mlsanity.com/paper/2608.09925/) (2026)
- [Language Models Are "Insecure" Reporters](https://mlsanity.com/paper/2609.36139/) (2026)
- [A Trust Layer for Agent Evaluation](https://mlsanity.com/paper/2610.07274/) (2026)
- [DecepEval: A Benchmark for Evaluating Deception in LLM Agents](https://mlsanity.com/paper/2610.07967/) (2026)

## Abstract (arXiv)

Previous works on Large Language Models (LLMs) have mainly focused on evaluating their helpfulness or harmlessness. However, honesty, another crucial alignment criterion, has received relatively less attention. Dishonest behaviors in LLMs, such as spreading misinformation and defrauding users, present severe risks that intensify as these models approach superintelligent levels. Enhancing honesty in LLMs addresses critical limitations and helps uncover latent capabilities that are not readily expressed. This underscores the urgent need for reliable methods and benchmarks to effectively ensure and evaluate the honesty of LLMs. In this paper, we introduce BeHonest, a pioneering benchmark specifically designed to assess honesty in LLMs comprehensively. BeHonest evaluates three essential aspects of honesty: awareness of knowledge boundaries, avoidance of deceit, and consistency in responses. Building on this foundation, we designed 10 scenarios to evaluate and analyze 9 popular LLMs on the market, including both closed-source and open-source models from different model families with varied model sizes. Our findings indicate that there is still significant room for improvement in the honesty of LLMs. We encourage the AI community to prioritize honesty alignment in these models, which can harness their full potential to benefit society while preventing them from causing harm through deception or inconsistency. Our benchmark and code can be found at: urlhttps://github.com/GAIR-NLP/BeHonest.