# The Many Dimensions of Truthfulness: Crowdsourcing Misinformation Assessments on a Multidimensional Scale

arXiv: https://arxiv.org/abs/2108.01222 · HTML: https://mlsanity.com/paper/2108.01222/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Aug 2021.

## Built on by (most cited first, 5 of 5)

- [Debate-to-Detect: Reformulating Misinformation Detection as a Real-World Debate with Large Language Models](https://mlsanity.com/paper/2505.18596/) (2025)
- [Assessing the Potential of Generative Agents in Crowdsourced Fact-Checking](https://mlsanity.com/paper/2504.19940/) (2025)
- [Efficiency and Effectiveness of LLM-Based Summarization of Evidence in Crowdsourced Fact-Checking](https://mlsanity.com/paper/2501.18265/) (2025)
- [Adversarial and Safely Scaled Question Generation](https://mlsanity.com/paper/2210.09467/) (2022)
- [Beyond Seeing Is Believing: On Crowdsourced Detection of Audiovisual Deepfakes](https://mlsanity.com/paper/2605.04797/) (2026)

## Abstract (arXiv)

Recent work has demonstrated the viability of using crowdsourcing as a tool for evaluating the truthfulness of public statements. Under certain conditions such as: (1) having a balanced set of workers with different backgrounds and cognitive abilities; (2) using an adequate set of mechanisms to control the quality of the collected data; and (3) using a coarse grained assessment scale, the crowd can provide reliable identification of fake news. However, fake news are a subtle matter: statements can be just biased ("cherrypicked"), imprecise, wrong, etc. and the unidimensional truth scale used in existing work cannot account for such differences. In this paper we propose a multidimensional notion of truthfulness and we ask the crowd workers to assess seven different dimensions of truthfulness selected based on existing literature: Correctness, Neutrality, Comprehensibility, Precision, Completeness, Speaker's Trustworthiness, and Informativeness. We deploy a set of quality control mechanisms to ensure that the thousands of assessments collected on 180 publicly available fact-checked statements distributed over two datasets are of adequate quality, including a custom search engine used by the crowd workers to find web pages supporting their truthfulness assessments. A comprehensive analysis of crowdsourced judgments shows that: (1) the crowdsourced assessments are reliable when compared to an expert-provided gold standard; (2) the proposed dimensions of truthfulness capture independent pieces of information; (3) the crowdsourcing task can be easily learned by the workers; and (4) the resulting assessments provide a useful basis for a more complete estimation of statement truthfulness.