# Evidence-Grounded Multimodal Misinformation Detection with Attention-Based GNNs

arXiv: https://arxiv.org/abs/2505.18221 · HTML: https://mlsanity.com/paper/2505.18221/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 39 references that are in the map.
- First posted on arXiv 23 May 2025.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification](https://mlsanity.com/paper/2009.03509/) (2020)
- [Can LLM-Generated Misinformation Be Detected?](https://mlsanity.com/paper/2309.13788/) (2023)
- [GCAN: Graph-aware Co-Attention Networks for Explainable Fake News Detection on Social Media](https://mlsanity.com/paper/2004.11648/) (2020)
- [NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media](https://mlsanity.com/paper/2104.05893/) (2021)
- [SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection](https://mlsanity.com/paper/2403.03170/) (2024)
- [Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resources](https://mlsanity.com/paper/2112.00061/) (2021)
- [MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMs](https://mlsanity.com/paper/2406.08772/) (2024)
- [FactKG: Fact Verification via Reasoning on Knowledge Graphs](https://mlsanity.com/paper/2305.06590/) (2023)
- [Explainable Claim Verification via Knowledge-Grounded Reasoning with Large Language Models](https://mlsanity.com/paper/2310.05253/) (2023)
- [LEMMA: Towards LVLM-Enhanced Multimodal Misinformation Detection with External Knowledge Augmentation](https://mlsanity.com/paper/2402.11943/) (2024)
- [COVE: COntext and VEracity prediction for out-of-context images](https://mlsanity.com/paper/2502.01194/) (2025)
- [Similarity over Factuality: Are we making progress on multimodal out-of-context misinformation detection?](https://mlsanity.com/paper/2407.13488/) (2024)
- [IA-GCN: Interpretable Attention based Graph Convolutional Network for Disease prediction](https://mlsanity.com/paper/2103.15587/) (2021)
- [Fact or Fiction? Improving Fact Verification with Knowledge Graphs through Simplified Subgraph Retrievals](https://mlsanity.com/paper/2408.07453/) (2024)

## Built on by (most cited first, 2 of 2)

- [Multimedia Verification Through Multi-Agent Deep Research Multimodal Large Language Models](https://mlsanity.com/paper/2507.04410/) (2025)
- [NepOOC-M: Bilingual Nepali-English Benchmark and Comparative Analysis of Multimodal Architectures for OOC Detection](https://mlsanity.com/paper/2608.19212/) (2026)

## Abstract (arXiv)

Multimodal out-of-context (OOC) misinformation is misinformation that repurposes real images with unrelated or misleading captions. Detecting such misinformation is challenging because it requires resolving the context of the claim before checking for misinformation. Many current methods, including LLMs and LVLMs, do not perform this contextualization step. LLMs hallucinate in absence of context or parametric knowledge. In this work, we propose a graph-based method that evaluates the consistency between the image and the caption by constructing two graph representations: an evidence graph, derived from online textual evidence, and a claim graph, from the claim in the caption. Using graph neural networks (GNNs) to encode and compare these representations, our framework then evaluates the truthfulness of image-caption pairs. We create datasets for our graph-based method, evaluate and compare our baseline model against popular LLMs on the misinformation detection task. Our method scores 93.05% detection accuracy on the evaluation set and outperforms the second-best performing method (an LLM) by 2.82%, making a case for smaller and task-specific methods.