# TruthEval: A Dataset to Evaluate LLM Truthfulness and Reliability

arXiv: https://arxiv.org/abs/2406.01855 · HTML: https://mlsanity.com/paper/2406.01855/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 29 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Builds on (most cited first, 10 of 10)

- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- ["Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection](https://mlsanity.com/paper/1705.00648/) (2017)
- [Entity-level Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/2102.09130/) (2021)
- [Assessing The Factual Accuracy of Generated Text](https://mlsanity.com/paper/1905.13322/) (2019)
- [A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking](https://mlsanity.com/paper/1911.01214/) (2019)
- [Improving Factual Consistency of Abstractive Summarization via Question Answering](https://mlsanity.com/paper/2105.04623/) (2021)
- [CheckThat! at CLEF 2020: Enabling the Automatic Identification and Verification of Claims in Social Media](https://mlsanity.com/paper/2001.08546/) (2020)
- [Reliability Check: An Analysis of GPT-3's Response to Sensitive Topics and Prompt Wording](https://mlsanity.com/paper/2306.06199/) (2023)

## Built on by (most cited first, 7 of 7)

- [The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs](https://mlsanity.com/paper/2506.11094/) (2025)
- [When Truthful Representations Flip Under Deceptive Instructions?](https://mlsanity.com/paper/2507.22149/) (2025)
- [Assessing Language Models' Worldview for Fiction Generation](https://mlsanity.com/paper/2408.07904/) (2024)
- [Red Teaming Large Reasoning Models](https://mlsanity.com/paper/2512.00412/) (2025)
- [Learning Uncertainty from Sequential Internal Dispersion in Large Language Models](https://mlsanity.com/paper/2604.15741/) (2026)
- [How Good LLM-Generated Password Policies Are?](https://mlsanity.com/paper/2506.08320/) (2025)
- [The Flaw of Averages: Measuring Benchmark-Level Distributional Robustness](https://mlsanity.com/paper/2509.25671/) (2025)

## Abstract (arXiv)

Large Language Model (LLM) evaluation is currently one of the most important areas of research, with existing benchmarks proving to be insufficient and not completely representative of LLMs' various capabilities. We present a curated collection of challenging statements on sensitive topics for LLM benchmarking called TruthEval. These statements were curated by hand and contain known truth values. The categories were chosen to distinguish LLMs' abilities from their stochastic nature. We perform some initial analyses using this dataset and find several instances of LLMs failing in simple tasks showing their inability to understand simple questions.