# E2LVLM:Evidence-Enhanced Large Vision-Language Model for Multimodal Out-of-Context Misinformation Detection

arXiv: https://arxiv.org/abs/2502.10455 · HTML: https://mlsanity.com/paper/2502.10455/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 32 of its 49 references that are in the map.
- First posted on arXiv 12 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [GeoChat: Grounded Large Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2311.15826/) (2023)
- [GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction](https://mlsanity.com/paper/2305.18752/) (2023)
- [Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detection](https://mlsanity.com/paper/2312.16649/) (2023)
- [Hallucination Augmented Contrastive Learning for Multimodal Large Language Model](https://mlsanity.com/paper/2312.06968/) (2023)
- [NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media](https://mlsanity.com/paper/2104.05893/) (2021)
- [LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge](https://mlsanity.com/paper/2311.11860/) (2023)
- [Detecting Multimedia Generated by Large AI Models: A Survey](https://mlsanity.com/paper/2402.00045/) (2024)
- [SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection](https://mlsanity.com/paper/2403.03170/) (2024)
- [Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resources](https://mlsanity.com/paper/2112.00061/) (2021)
- [Detecting and Grounding Multi-Modal Media Manipulation and Beyond](https://mlsanity.com/paper/2309.14203/) (2023)
- [Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors](https://mlsanity.com/paper/2305.11159/) (2023)
- [VERITE: A Robust Benchmark for Multimodal Misinformation Detection Accounting for Unimodal Bias](https://mlsanity.com/paper/2304.14133/) (2023)
- [FKA-Owl: Advancing Multimodal Fake News Detection through Knowledge-Augmented LVLMs](https://mlsanity.com/paper/2403.01988/) (2024)
- [SAFE: Self-Attentive Function Embeddings for Binary Similarity](https://mlsanity.com/paper/1811.05296/) (2018)
- [Support or Refute: Analyzing the Stance of Evidence to Detect Out-of-Context Mis- and Disinformation](https://mlsanity.com/paper/2311.01766/) (2023)
- [Multimodal Misinformation Detection using Large Vision-Language Models](https://mlsanity.com/paper/2407.14321/) (2024)
- [RED-DOT: Multimodal Fact-checking via Relevant Evidence Detection](https://mlsanity.com/paper/2311.09939/) (2023)
- [CoTKR: Chain-of-Thought Enhanced Knowledge Rewriting for Complex Knowledge Graph Question Answering](https://mlsanity.com/paper/2409.19753/) (2024)
- [LLaVA-VSD: Large Language-and-Vision Assistant for Visual Spatial Description](https://mlsanity.com/paper/2408.04957/) (2024)

## Built on by (most cited first, 7 of 7)

- [Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMs](https://mlsanity.com/paper/2601.19202/) (2026)
- [Drifting Away from Truth: GenAI-Driven News Diversity Challenges LVLM-Based Misinformation Detection](https://mlsanity.com/paper/2508.12711/) (2025)
- [The Paradigm Shift: A Comprehensive Survey on Large Vision Language Models for Multimodal Fake News Detection](https://mlsanity.com/paper/2601.15316/) (2026)
- [T^2Agent A Tool-augmented Multimodal Misinformation Detection Agent with Monte Carlo Tree Search](https://mlsanity.com/paper/2505.19768/) (2025)
- [What's Left Unsaid? Detecting and Correcting Misleading Omissions in Multimodal News Previews](https://mlsanity.com/paper/2601.05563/) (2026)
- [Evidence Triangulation for Multimodal Fact-Checking in the Wild](https://mlsanity.com/paper/2606.31367/) (2026)
- [SAFE-MR: Evidence Sufficiency Learning for Selective Multimodal Rumor Detection](https://mlsanity.com/paper/2610.06708/) (2026)

## Abstract (arXiv)

Recent studies in Large Vision-Language Models (LVLMs) have demonstrated impressive advancements in multimodal Out-of-Context (OOC) misinformation detection, discerning whether an authentic image is wrongly used in a claim. Despite their success, the textual evidence of authentic images retrieved from the inverse search is directly transmitted to LVLMs, leading to inaccurate or false information in the decision-making phase. To this end, we present E2LVLM, a novel evidence-enhanced large vision-language model by adapting textual evidence in two levels. First, motivated by the fact that textual evidence provided by external tools struggles to align with LVLMs inputs, we devise a reranking and rewriting strategy for generating coherent and contextually attuned content, thereby driving the aligned and effective behavior of LVLMs pertinent to authentic images. Second, to address the scarcity of news domain datasets with both judgment and explanation, we generate a novel OOC multimodal instruction-following dataset by prompting LVLMs with informative content to acquire plausible explanations. Further, we develop a multimodal instruction-tuning strategy with convincing explanations for beyond detection. This scheme contributes to E2LVLM for multimodal OOC misinformation detection and explanation. A multitude of experiments demonstrate that E2LVLM achieves superior performance than state-of-the-art methods, and also provides compelling rationales for judgments.