# Similarity over Factuality: Are we making progress on multimodal out-of-context misinformation detection?

arXiv: https://arxiv.org/abs/2407.13488 · HTML: https://mlsanity.com/paper/2407.13488/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 18 Jul 2024.

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [Visual News: Benchmark and Challenges in News Image Captioning](https://mlsanity.com/paper/2010.03743/) (2020)
- [NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media](https://mlsanity.com/paper/2104.05893/) (2021)
- [End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models](https://mlsanity.com/paper/2205.12487/) (2022)
- [Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forward](https://mlsanity.com/paper/2103.00484/) (2021)
- [SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection](https://mlsanity.com/paper/2403.03170/) (2024)
- [Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resources](https://mlsanity.com/paper/2112.00061/) (2021)
- [MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMs](https://mlsanity.com/paper/2406.08772/) (2024)
- [SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models](https://mlsanity.com/paper/2002.06652/) (2020)
- [Where Are the Facts? Searching for Fact-checked Information to Alleviate the Spread of Fake News](https://mlsanity.com/paper/2010.03159/) (2020)
- [Missing Counter-Evidence Renders NLP Fact-Checking Unrealistic for Misinformation](https://mlsanity.com/paper/2210.13865/) (2022)
- [VERITE: A Robust Benchmark for Multimodal Misinformation Detection Accounting for Unimodal Bias](https://mlsanity.com/paper/2304.14133/) (2023)
- [Multimodal Analytics for Real-world News using Measures of Cross-modal Entity Consistency](https://mlsanity.com/paper/2003.10421/) (2020)
- [Multilingual Previously Fact-Checked Claim Retrieval](https://mlsanity.com/paper/2305.07991/) (2023)
- [Deep Multimodal Image-Repurposing Detection](https://mlsanity.com/paper/1808.06686/) (2018)
- [Multimedia Semantic Integrity Assessment Using Joint Embedding Of Images And Text](https://mlsanity.com/paper/1707.01606/) (2017)
- [CheckThat! at CLEF 2020: Enabling the Automatic Identification and Verification of Claims in Social Media](https://mlsanity.com/paper/2001.08546/) (2020)
- [Synthetic Misinformers: Generating and Combating Multimodal Misinformation](https://mlsanity.com/paper/2303.01217/) (2023)
- [Support or Refute: Analyzing the Stance of Evidence to Detect Out-of-Context Mis- and Disinformation](https://mlsanity.com/paper/2311.01766/) (2023)
- [RED-DOT: Multimodal Fact-checking via Relevant Evidence Detection](https://mlsanity.com/paper/2311.09939/) (2023)
- [Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation](https://mlsanity.com/paper/2112.08594/) (2021)
- [Credible, Unreliable or Leaked?: Evidence Verification for Enhanced Automated Fact-checking](https://mlsanity.com/paper/2404.18971/) (2024)
- [MMSys'21 Grand Challenge on Detecting Cheapfakes](https://mlsanity.com/paper/2107.05297/) (2021)
- [Learning Domain-Invariant Features for Out-of-Context News Detection](https://mlsanity.com/paper/2406.07430/) (2024)

## Built on by (most cited first, 11 of 11)

- [DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts](https://mlsanity.com/paper/2412.10510/) (2024)
- [AVerImaTeC: A Dataset for Automatic Verification of Image-Text Claims with Evidence from the Web](https://mlsanity.com/paper/2505.17978/) (2025)
- [M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset](https://mlsanity.com/paper/2510.23508/) (2025)
- [RAMA: Retrieval-Augmented Multi-Agent Framework for Misinformation Detection in Multimodal Fact-Checking](https://mlsanity.com/paper/2507.09174/) (2025)
- [Evidence-Grounded Multimodal Misinformation Detection with Attention-Based GNNs](https://mlsanity.com/paper/2505.18221/) (2025)
- ["Humor, Art, or Misinformation?": A Multimodal Dataset for Intent-Aware Synthetic Image Detection](https://mlsanity.com/paper/2508.20670/) (2025)
- [The Automatic Verification of Image-Text Claims (AVerImaTeC) Shared Task](https://mlsanity.com/paper/2602.11221/) (2026)
- [Latent Reconstruction from Generated Data for Multimodal Misinformation Detection](https://mlsanity.com/paper/2504.06010/) (2025)
- [From Cheap Fakes to Pure Synthesis: Addressing the New Era of T2V Fake News Videos](https://mlsanity.com/paper/2608.06732/) (2026)
- [Evidence Triangulation for Multimodal Fact-Checking in the Wild](https://mlsanity.com/paper/2606.31367/) (2026)
- [MIC: Explaining Image-Claim Inconsistencies in AI-Generated Multimodal Misinformation](https://mlsanity.com/paper/2609.33441/) (2026)

## Abstract (arXiv)

Out-of-context (OOC) misinformation poses a significant challenge in multimodal fact-checking, where images are paired with texts that misrepresent their original context to support false narratives. Recent research in evidence-based OOC detection has seen a trend towards increasingly complex architectures, incorporating Transformers, foundation models, and large language models. In this study, we introduce a simple yet robust baseline, which assesses MUltimodal SimilaritiEs (MUSE), specifically the similarity between image-text pairs and external image and text evidence. Our results demonstrate that MUSE, when used with conventional classifiers like Decision Tree, Random Forest, and Multilayer Perceptron, can compete with and even surpass the state-of-the-art on the NewsCLIPpings and VERITE datasets. Furthermore, integrating MUSE in our proposed "Attentive Intermediate Transformer Representations" (AITR) significantly improved performance, by 3.3% and 7.5% on NewsCLIPpings and VERITE, respectively. Nevertheless, the success of MUSE, relying on surface-level patterns and shortcuts, without examining factuality and logical inconsistencies, raises critical questions about how we define the task, construct datasets, collect external evidence and overall, how we assess progress in the field. We release our code at: https://github.com/stevejpapad/outcontext-misinfo-progress