# CMIE: Combining MLLM Insights with External Evidence for Explainable Out-of-Context Misinformation Detection

arXiv: https://arxiv.org/abs/2505.23449 · HTML: https://mlsanity.com/paper/2505.23449/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 30 of its 34 references that are in the map.
- First posted on arXiv 29 May 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark](https://mlsanity.com/paper/2402.04788/) (2024)
- [Multimodal Foundation Models: From Specialists to General-Purpose Assistants](https://mlsanity.com/paper/2309.10020/) (2023)
- [Can LLM-Generated Misinformation Be Detected?](https://mlsanity.com/paper/2309.13788/) (2023)
- [NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media](https://mlsanity.com/paper/2104.05893/) (2021)
- [The science of fake news](https://mlsanity.com/paper/2307.07903/) (2023)
- [Towards LLM-based Fact Verification on News Claims with a Hierarchical Step-by-Step Prompting Method](https://mlsanity.com/paper/2310.00305/) (2023)
- [Efficient Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2405.10739/) (2024)
- [Combating Misinformation in the Age of LLMs: Opportunities and Challenges](https://mlsanity.com/paper/2311.05656/) (2023)
- [SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection](https://mlsanity.com/paper/2403.03170/) (2024)
- [Bad Actor, Good Advisor: Exploring the Role of Large Language Models in Fake News Detection](https://mlsanity.com/paper/2309.12247/) (2023)
- [Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resources](https://mlsanity.com/paper/2112.00061/) (2021)
- [FakeSV: A Multimodal Benchmark with Rich Social Context for Fake News Detection on Short Video Platforms](https://mlsanity.com/paper/2211.10973/) (2022)
- [VERITE: A Robust Benchmark for Multimodal Misinformation Detection Accounting for Unimodal Bias](https://mlsanity.com/paper/2304.14133/) (2023)
- [Explainable Claim Verification via Knowledge-Grounded Reasoning with Large Language Models](https://mlsanity.com/paper/2310.05253/) (2023)
- [Language Models Hallucinate, but May Excel at Fact Verification](https://mlsanity.com/paper/2310.14564/) (2023)
- [Multimodal Analytics for Real-world News using Measures of Cross-modal Entity Consistency](https://mlsanity.com/paper/2003.10421/) (2020)
- [Towards Fine-Grained Reasoning for Fake News Detection](https://mlsanity.com/paper/2110.15064/) (2021)
- [LEMMA: Towards LVLM-Enhanced Multimodal Misinformation Detection with External Knowledge Augmentation](https://mlsanity.com/paper/2402.11943/) (2024)
- [COSMOS: Catching Out-of-Context Misinformation with Self-Supervised Learning](https://mlsanity.com/paper/2101.06278/) (2021)
- [Cross-modal Contrastive Learning for Multimodal Fake News Detection](https://mlsanity.com/paper/2302.14057/) (2023)
- [Synthetic Misinformers: Generating and Combating Multimodal Misinformation](https://mlsanity.com/paper/2303.01217/) (2023)
- [Support or Refute: Analyzing the Stance of Evidence to Detect Out-of-Context Mis- and Disinformation](https://mlsanity.com/paper/2311.01766/) (2023)
- [Customized large language models can outperform Community Notes in correcting misinformation](https://mlsanity.com/paper/2403.11169/) (2024)
- [Are Large Language Models Good Fact Checkers: A Preliminary Study](https://mlsanity.com/paper/2311.17355/) (2023)
- [Two Heads Are Better Than One: Integrating Knowledge from Knowledge Graphs and Large Language Models for Entity Alignment](https://mlsanity.com/paper/2401.16960/) (2024)

## Built on by (most cited first, 6 of 6)

- [Recursive Self-Evolving Agents via Held-Out Selection](https://mlsanity.com/paper/2606.28374/) (2026)
- [Drifting Away from Truth: GenAI-Driven News Diversity Challenges LVLM-Based Misinformation Detection](https://mlsanity.com/paper/2508.12711/) (2025)
- [The Paradigm Shift: A Comprehensive Survey on Large Vision Language Models for Multimodal Fake News Detection](https://mlsanity.com/paper/2601.15316/) (2026)
- [DiFaR: Enhancing Multimodal Misinformation Detection with Diverse, Factual, and Relevant Rationales](https://mlsanity.com/paper/2508.10444/) (2025)
- [Multimodal Climate Disinformation Detection: Integrating Vision-Language Models with External Knowledge Sources](https://mlsanity.com/paper/2601.16108/) (2026)
- [Verification-Notebook Learning for Source-Aware Multimodal Misinformation Detection](https://mlsanity.com/paper/2607.23581/) (2026)

## Abstract (arXiv)

Multimodal large language models (MLLMs) have demonstrated impressive capabilities in visual reasoning and text generation. While previous studies have explored the application of MLLM for detecting out-of-context (OOC) misinformation, our empirical analysis reveals two persisting challenges of this paradigm. Evaluating the representative GPT-4o model on direct reasoning and evidence augmented reasoning, results indicate that MLLM struggle to capture the deeper relationships-specifically, cases in which the image and text are not directly connected but are associated through underlying semantic links. Moreover, noise in the evidence further impairs detection accuracy. To address these challenges, we propose CMIE, a novel OOC misinformation detection framework that incorporates a Coexistence Relationship Generation (CRG) strategy and an Association Scoring (AS) mechanism. CMIE identifies the underlying coexistence relationships between images and text, and selectively utilizes relevant evidence to enhance misinformation detection. Experimental results demonstrate that our approach outperforms existing methods.