# Misleading through Inconsistency: A Benchmark for Political Inconsistencies Detection

arXiv: https://arxiv.org/abs/2505.19191 · HTML: https://mlsanity.com/paper/2505.19191/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 57 references that are in the map.
- First posted on arXiv 25 May 2025.

## Builds on (most cited first, 11 of 11)

- [Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics](https://mlsanity.com/paper/2104.13346/) (2021)
- [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://mlsanity.com/paper/2112.08542/) (2021)
- [LM vs LM: Detecting Factual Errors via Cross Examination](https://mlsanity.com/paper/2305.13281/) (2023)
- [ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts](https://mlsanity.com/paper/2110.01799/) (2021)
- [Stance Detection on Social Media: State of the Art and Trends](https://mlsanity.com/paper/2006.03644/) (2020)
- [DocNLI: A Large-scale Dataset for Document-level Natural Language Inference](https://mlsanity.com/paper/2106.09449/) (2021)
- [RCOT: Detecting and Rectifying Factual Inconsistency in Reasoning by Reversing Chain-of-Thought](https://mlsanity.com/paper/2305.11499/) (2023)
- [X-Stance: A Multilingual Multi-Target Dataset for Stance Detection](https://mlsanity.com/paper/2003.08385/) (2020)
- [LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond](https://mlsanity.com/paper/2305.14540/) (2023)
- [Fast and Accurate Factual Inconsistency Detection Over Long Documents](https://mlsanity.com/paper/2310.13189/) (2023)
- [Prospects for inconsistency detection using large language models and sheaves](https://mlsanity.com/paper/2401.16713/) (2024)

## Built on by (most cited first, 1 of 1)

- [On Finding Inconsistencies in Documents](https://mlsanity.com/paper/2512.18601/) (2025)

## Abstract (arXiv)

Inconsistent political statements represent a form of misinformation. They erode public trust and pose challenges to accountability, when left unnoticed. Detecting inconsistencies automatically could support journalists in asking clarification questions, thereby helping to keep politicians accountable. We propose the Inconsistency detection task and develop a scale of inconsistency types to prompt NLP-research in this direction. To provide a resource for detecting inconsistencies in a political domain, we present a dataset of 698 human-annotated pairs of political statements with explanations of the annotators' reasoning for 237 samples. The statements mainly come from voting assistant platforms such as Wahl-O-Mat in Germany and Smartvote in Switzerland, reflecting real-world political issues. We benchmark Large Language Models (LLMs) on our dataset and show that in general, they are as good as humans at detecting inconsistencies, and might be even better than individual humans at predicting the crowd-annotated ground-truth. However, when it comes to identifying fine-grained inconsistency types, none of the model have reached the upper bound of performance (due to natural labeling variation), thus leaving room for improvement. We make our dataset and code publicly available.