# HintsOfTruth: A Multimodal Checkworthiness Detection Dataset with Real and Synthetic Claims

arXiv: https://arxiv.org/abs/2502.11753 · HTML: https://mlsanity.com/paper/2502.11753/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 87 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Builds on (most cited first, 30 of 42)

- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [Automated Fact-Checking for Assisting Human Fact-Checkers](https://mlsanity.com/paper/2103.07769/) (2021)
- [AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web](https://mlsanity.com/paper/2305.13117/) (2023)
- [On the Risk of Misinformation Pollution with Large Language Models](https://mlsanity.com/paper/2305.13661/) (2023)
- [Factuality Challenges in the Era of Large Language Models](https://mlsanity.com/paper/2310.05189/) (2023)
- [Stability](https://mlsanity.com/paper/1310.0150/) (2013)
- [NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media](https://mlsanity.com/paper/2104.05893/) (2021)
- [Detecting Multimedia Generated by Large AI Models: A Survey](https://mlsanity.com/paper/2402.00045/) (2024)
- [Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resources](https://mlsanity.com/paper/2112.00061/) (2021)
- [Generating Literal and Implied Subquestions to Fact-check Complex Claims](https://mlsanity.com/paper/2205.06938/) (2022)
- [MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMs](https://mlsanity.com/paper/2406.08772/) (2024)
- [Towards Automated Factchecking: Developing an Annotation Schema and Benchmark for Consistent Automated Claim Detection](https://mlsanity.com/paper/1809.08193/) (2018)
- [From Pixels to Prose: A Large Dataset of Dense Image Captions](https://mlsanity.com/paper/2406.10328/) (2024)
- [Multimodal Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2305.13507/) (2023)
- [Fact-Checking Meets Fauxtography: Verifying Claims About Images](https://mlsanity.com/paper/1908.11722/) (2019)
- [Overview of CheckThat! 2020: Automatic Identification and Verification of Claims in Social Media](https://mlsanity.com/paper/2007.07997/) (2020)
- [Missing Counter-Evidence Renders NLP Fact-Checking Unrealistic for Misinformation](https://mlsanity.com/paper/2210.13865/) (2022)
- [The Larger the Better? Improved LLM Code-Generation via Budget Reallocation](https://mlsanity.com/paper/2404.00725/) (2024)
- [Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. Task 1: Check-Worthiness](https://mlsanity.com/paper/1808.05542/) (2018)
- [VERITE: A Robust Benchmark for Multimodal Misinformation Detection Accounting for Unimodal Bias](https://mlsanity.com/paper/2304.14133/) (2023)
- [Multilingual Previously Fact-Checked Claim Retrieval](https://mlsanity.com/paper/2305.07991/) (2023)
- [AMMeBa: A Large-Scale Survey and Dataset of Media-Based Misinformation In-The-Wild](https://mlsanity.com/paper/2405.11697/) (2024)
- [Evidence-backed Fact Checking using RAG and Few-Shot In-Context Learning with LLMs](https://mlsanity.com/paper/2408.12060/) (2024)

## Built on by (most cited first, 4 of 4)

- [VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking](https://mlsanity.com/paper/2601.08611/) (2026)
- [Multimodal Claim Extraction for Fact-Checking](https://mlsanity.com/paper/2604.16311/) (2026)
- [ClaimPT: A Portuguese Dataset of Annotated Claims in News Articles](https://mlsanity.com/paper/2601.19490/) (2026)
- [UNMATCH: Selective Unbalanced Token-Patch Matching for Forensic Image-Claim Verification](https://mlsanity.com/paper/2609.31766/) (2026)

## Abstract (arXiv)

Misinformation can be countered with fact-checking, but the process is costly and slow. Identifying checkworthy claims is the first step, where automation can help scale fact-checkers' efforts. However, detection methods struggle with content that is (1) multimodal, (2) from diverse domains, and (3) synthetic. We introduce HintsOfTruth, a public dataset for multimodal checkworthiness detection with 27K real-world and synthetic image/claim pairs. The mix of real and synthetic data makes this dataset unique and ideal for benchmarking detection methods. We compare fine-tuned and prompted Large Language Models (LLMs). We find that well-configured lightweight text-based encoders perform comparably to multimodal models but the former only focus on identifying non-claim-like content. Multimodal LLMs can be more accurate but come at a significant computational cost, making them impractical for large-scale applications. When faced with synthetic data, multimodal models perform more robustly.