# Piecing It All Together: Verifying Multi-Hop Multimodal Claims

arXiv: https://arxiv.org/abs/2411.09547 · HTML: https://mlsanity.com/paper/2411.09547/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 53 references that are in the map.
- First posted on arXiv 14 Nov 2024.

## Builds on (most cited first, 30 of 35)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [Visual Programming: Compositional visual reasoning without training](https://mlsanity.com/paper/2211.11559/) (2022)
- [Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models](https://mlsanity.com/paper/2402.17177/) (2024)
- [The Web as a Knowledge-base for Answering Complex Questions](https://mlsanity.com/paper/1803.06643/) (2018)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- ["Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection](https://mlsanity.com/paper/1705.00648/) (2017)
- [FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Media](https://mlsanity.com/paper/1809.01286/) (2018)
- [Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies](https://mlsanity.com/paper/2308.03188/) (2023)
- [HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification](https://mlsanity.com/paper/2011.03088/) (2020)
- [Constructing Datasets for Multi-hop Reading Comprehension Across Documents](https://mlsanity.com/paper/1710.06481/) (2017)
- [MultiModalQA: Complex Question Answering over Text, Tables and Images](https://mlsanity.com/paper/2104.06039/) (2021)
- [Fact-Checking Complex Claims with Program-Guided Reasoning](https://mlsanity.com/paper/2305.12744/) (2023)
- [NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media](https://mlsanity.com/paper/2104.05893/) (2021)
- [End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models](https://mlsanity.com/paper/2205.12487/) (2022)
- [Automated Fact Checking: Task formulations, methods and future directions](https://mlsanity.com/paper/1806.07687/) (2018)
- [Multimodal Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2305.13507/) (2023)
- [Fact-Checking Meets Fauxtography: Verifying Claims About Images](https://mlsanity.com/paper/1908.11722/) (2019)
- [LLM-as-a-Coauthor: Can Mixed Human-Written and Machine-Generated Text Be Detected?](https://mlsanity.com/paper/2401.05952/) (2024)
- [Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges](https://mlsanity.com/paper/2408.08946/) (2024)
- [Can Large Language Models Identify Authorship?](https://mlsanity.com/paper/2403.08213/) (2024)
- [Explainable Claim Verification via Knowledge-Grounded Reasoning with Large Language Models](https://mlsanity.com/paper/2310.05253/) (2023)
- [Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Models](https://mlsanity.com/paper/2310.13671/) (2023)

## Built on by (most cited first, 12 of 12)

- [Can Multimodal LLMs Perform Time Series Anomaly Detection?](https://mlsanity.com/paper/2502.17812/) (2025)
- [FCMR: Robust Evaluation of Financial Cross-Modal Multi-Hop Reasoning](https://mlsanity.com/paper/2412.12567/) (2024)
- [VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking](https://mlsanity.com/paper/2601.08611/) (2026)
- [SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory](https://mlsanity.com/paper/2605.15710/) (2026)
- [Domain Specific Benchmarks for Evaluating Multimodal Large Language Models](https://mlsanity.com/paper/2506.12958/) (2025)
- [CRIT: Graph-Based Automatic Data Synthesis to Enhance Cross-Modal Multi-Hop Reasoning](https://mlsanity.com/paper/2604.01634/) (2026)
- [LiveFact: A Dynamic, Time-Aware Benchmark for LLM-Driven Fake News Detection](https://mlsanity.com/paper/2604.04815/) (2026)
- [M2-Verify: A Large-Scale Multidomain Benchmark for Checking Multimodal Claim Consistency](https://mlsanity.com/paper/2604.01306/) (2026)
- [Multimodal Claim Extraction for Fact-Checking](https://mlsanity.com/paper/2604.16311/) (2026)
- [MEVER: Multi-Modal and Explainable Claim Verification with Graph-based Evidence Retrieval](https://mlsanity.com/paper/2602.10023/) (2026)
- [ToolSciVer: Multimodal Scientific Claim Verification with Visual Tool Augmented Reinforcement Learning](https://mlsanity.com/paper/2607.16131/) (2026)
- [ASTRA - Agentic System for Ticket Resolution and Analysis](https://mlsanity.com/paper/2608.28790/) (2026)

## Abstract (arXiv)

Existing claim verification datasets often do not require systems to perform complex reasoning or effectively interpret multimodal evidence. To address this, we introduce a new task: multi-hop multimodal claim verification. This task challenges models to reason over multiple pieces of evidence from diverse sources, including text, images, and tables, and determine whether the combined multimodal evidence supports or refutes a given claim. To study this task, we construct MMCV, a large-scale dataset comprising 15k multi-hop claims paired with multimodal evidence, generated and refined using large language models, with additional input from human feedback. We show that MMCV is challenging even for the latest state-of-the-art multimodal large language models, especially as the number of reasoning hops increases. Additionally, we establish a human performance benchmark on a subset of MMCV. We hope this dataset and its evaluation task will encourage future research in multimodal multi-hop claim verification.