# COVE: COntext and VEracity prediction for out-of-context images

arXiv: https://arxiv.org/abs/2502.01194 · HTML: https://mlsanity.com/paper/2502.01194/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 49 references that are in the map.
- First posted on arXiv 3 Feb 2025.

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Autoregressive Entity Retrieval](https://mlsanity.com/paper/2010.00904/) (2020)
- [Fact-Checking Complex Claims with Program-Guided Reasoning](https://mlsanity.com/paper/2305.12744/) (2023)
- [AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web](https://mlsanity.com/paper/2305.13117/) (2023)
- [Visual News: Benchmark and Challenges in News Image Captioning](https://mlsanity.com/paper/2010.03743/) (2020)
- [Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entities](https://mlsanity.com/paper/2302.11154/) (2023)
- [NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media](https://mlsanity.com/paper/2104.05893/) (2021)
- [SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection](https://mlsanity.com/paper/2403.03170/) (2024)
- [Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resources](https://mlsanity.com/paper/2112.00061/) (2021)
- [WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia](https://mlsanity.com/paper/2305.14292/) (2023)
- [DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts](https://mlsanity.com/paper/2412.10510/) (2024)
- [Multimodal Analytics for Real-world News using Measures of Cross-modal Entity Consistency](https://mlsanity.com/paper/2003.10421/) (2020)
- [Deep Multimodal Image-Repurposing Detection](https://mlsanity.com/paper/1808.06686/) (2018)
- [AMMeBa: A Large-Scale Survey and Dataset of Media-Based Misinformation In-The-Wild](https://mlsanity.com/paper/2405.11697/) (2024)
- [QACHECK: A Demonstration System for Question-Guided Multi-Hop Fact-Checking](https://mlsanity.com/paper/2310.07609/) (2023)
- [Support or Refute: Analyzing the Stance of Evidence to Detect Out-of-Context Mis- and Disinformation](https://mlsanity.com/paper/2311.01766/) (2023)
- [Generating Media Background Checks for Automated Source Critical Reasoning](https://mlsanity.com/paper/2409.00781/) (2024)
- [Credible, Unreliable or Leaked?: Evidence Verification for Enhanced Automated Fact-checking](https://mlsanity.com/paper/2404.18971/) (2024)

## Built on by (most cited first, 14 of 14)

- [M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset](https://mlsanity.com/paper/2510.23508/) (2025)
- [VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking](https://mlsanity.com/paper/2601.08611/) (2026)
- [T^2Agent A Tool-augmented Multimodal Misinformation Detection Agent with Monte Carlo Tree Search](https://mlsanity.com/paper/2505.19768/) (2025)
- [GETReason: Enhancing Image Context Extraction through Hierarchical Multi-Agent Reasoning](https://mlsanity.com/paper/2505.21863/) (2025)
- [Evidence-Grounded Multimodal Misinformation Detection with Attention-Based GNNs](https://mlsanity.com/paper/2505.18221/) (2025)
- [The Synthetic Media Shift: Tracking the Rise, Virality, and Detectability of AI-Generated Multimodal Misinformation](https://mlsanity.com/paper/2604.15372/) (2026)
- [ChartAttack: Testing the Vulnerability of LLMs to Malicious Prompting in Chart Generation](https://mlsanity.com/paper/2601.12983/) (2026)
- [NewsRECON: News Article Retrieval for Image Contextualization](https://mlsanity.com/paper/2601.14121/) (2026)
- [Knowledge-Intensive Video Generation](https://mlsanity.com/paper/2606.01285/) (2026)
- [The Warrant Gap: Claim-Conditioned Re-scoring for Fact-Checking](https://mlsanity.com/paper/2606.24627/) (2026)
- [NepOOC-M: Bilingual Nepali-English Benchmark and Comparative Analysis of Multimodal Architectures for OOC Detection](https://mlsanity.com/paper/2608.19212/) (2026)
- [RCMN: Understanding Misleadingness in Influential Public Discourse](https://mlsanity.com/paper/2608.27358/) (2026)
- [UNMATCH: Selective Unbalanced Token-Patch Matching for Forensic Image-Claim Verification](https://mlsanity.com/paper/2609.31766/) (2026)
- [MIC: Explaining Image-Claim Inconsistencies in AI-Generated Multimodal Misinformation](https://mlsanity.com/paper/2609.33441/) (2026)

## Abstract (arXiv)

Images taken out of their context are the most prevalent form of multimodal misinformation. Debunking them requires (1) providing the true context of the image and (2) checking the veracity of the image's caption. However, existing automated fact-checking methods fail to tackle both objectives explicitly. In this work, we introduce COVE, a new method that predicts first the true COntext of the image and then uses it to predict the VEracity of the caption. COVE beats the SOTA context prediction model on all context items, often by more than five percentage points. It is competitive with the best veracity prediction models on synthetic data and outperforms them on real-world data, showing that it is beneficial to combine the two tasks sequentially. Finally, we conduct a human study that reveals that the predicted context is a reusable and interpretable artifact to verify new out-of-context captions for the same image. Our code and data are made available.