# LEMMA: Towards LVLM-Enhanced Multimodal Misinformation Detection with External Knowledge Augmentation

arXiv: https://arxiv.org/abs/2402.11943 · HTML: https://mlsanity.com/paper/2402.11943/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 40 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Builds on (most cited first, 15 of 15)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios](https://mlsanity.com/paper/2307.13528/) (2023)
- [LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents](https://mlsanity.com/paper/2311.05437/) (2023)
- [Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers](https://mlsanity.com/paper/2311.09000/) (2023)
- [r/Fakeddit: A New Multimodal Benchmark Dataset for Fine-grained Fake News Detection](https://mlsanity.com/paper/1911.03854/) (2019)
- [On the Origins of Memes by Means of Fringe Web Communities](https://mlsanity.com/paper/1805.12512/) (2018)
- [RAG-Fusion: a New Take on Retrieval-Augmented Generation](https://mlsanity.com/paper/2402.03367/) (2024)
- [FLEEK: Factual Error Detection and Correction with Evidence Retrieved from External Knowledge](https://mlsanity.com/paper/2310.17119/) (2023)
- [FNR: A Similarity and Transformer-Based Approach to Detect Multi-Modal Fake News in Social Media](https://mlsanity.com/paper/2112.01131/) (2021)

## Built on by (most cited first, 28 of 28)

- [MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models](https://mlsanity.com/paper/2410.08182/) (2024)
- [Let Silence Speak: Enhancing Fake News Detection with Generated Comments from Large Language Models](https://mlsanity.com/paper/2405.16631/) (2024)
- [RAEmoLLM: Retrieval Augmented LLMs for Cross-Domain Misinformation Detection Using In-Context Learning Based on Emotional Information](https://mlsanity.com/paper/2406.11093/) (2024)
- [Multimodal Misinformation Detection by Learning from Synthetic Data with Multimodal LLMs](https://mlsanity.com/paper/2409.19656/) (2024)
- [GAMED: Knowledge Adaptive Multi-Experts Decoupling for Multimodal Fake News Detection](https://mlsanity.com/paper/2412.12164/) (2024)
- [TRUST-VL: An Explainable News Assistant for General Multimodal Misinformation Detection](https://mlsanity.com/paper/2509.04448/) (2025)
- [XFacta: Contemporary, Real-World Dataset and Evaluation for Multimodal Misinformation Detection with Multimodal LLMs](https://mlsanity.com/paper/2508.09999/) (2025)
- [CMIE: Combining MLLM Insights with External Evidence for Explainable Out-of-Context Misinformation Detection](https://mlsanity.com/paper/2505.23449/) (2025)
- [A Survey on Proactive Defense Strategies Against Misinformation in Large Language Models](https://mlsanity.com/paper/2507.05288/) (2025)
- [The Paradigm Shift: A Comprehensive Survey on Large Vision Language Models for Multimodal Fake News Detection](https://mlsanity.com/paper/2601.15316/) (2026)
- [Fake News Detection: Comparative Evaluation of BERT-like Models and Large Language Models with Generative AI-Annotated Data](https://mlsanity.com/paper/2412.14276/) (2024)
- [Seeing Through Deception: Uncovering Misleading Creator Intent in Multimodal News with Vision-Language Models](https://mlsanity.com/paper/2505.15489/) (2025)
- [Can Large Vision-Language Models Understand Multimodal Sarcasm?](https://mlsanity.com/paper/2508.03654/) (2025)
- [What's Left Unsaid? Detecting and Correcting Misleading Omissions in Multimodal News Previews](https://mlsanity.com/paper/2601.05563/) (2026)
- [Evidence-Grounded Multimodal Misinformation Detection with Attention-Based GNNs](https://mlsanity.com/paper/2505.18221/) (2025)
- [RW-Post: Auditable Evidence-Grounded Multimodal Fact-Checking in the Wild](https://mlsanity.com/paper/2605.10357/) (2026)
- [Beyond Retrieval: Improving Evidence Quality for LLM-based Multimodal Fact-Checking](https://mlsanity.com/paper/2505.03135/) (2025)
- [Augmented Vision-Language Models: A Systematic Review](https://mlsanity.com/paper/2507.22933/) (2025)
- [E-CaTCH: Event-Centric Cross-Modal Attention with Temporal Consistency and Class-Imbalance Handling for Misinformation Detection](https://mlsanity.com/paper/2508.11197/) (2025)
- [Effectiveness of Large Multimodal Models in Detecting Disinformation: Experimental Results](https://mlsanity.com/paper/2509.22377/) (2025)
- [MERIT: Modular Framework for Multimodal Misinformation Detection with Web-Grounded Reasoning](https://mlsanity.com/paper/2510.17590/) (2025)
- [Enhancing Multimodal Misinformation Detection by Replaying the Whole Story from Image Modality Perspective](https://mlsanity.com/paper/2511.06284/) (2025)
- [They Said Memes Were Harmless-We Found the Ones That Hurt: Decoding Jokes, Symbols, and Cultural References](https://mlsanity.com/paper/2602.03822/) (2026)
- [Benchmarking Deflection and Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2604.12033/) (2026)
- [Dynamic Analysis and Adaptive Discriminator for Fake News Detection](https://mlsanity.com/paper/2408.10883/) (2024)
- [RW-Post: Auditable Evidence-Grounded Multimodal Fact-Checking in the Wild](https://mlsanity.com/paper/2512.22933/) (2025)
- [Multimodal Climate Disinformation Detection: Integrating Vision-Language Models with External Knowledge Sources](https://mlsanity.com/paper/2601.16108/) (2026)
- [MOMENTA: Mixture-of-Experts Over Multimodal Embeddings with Neural Temporal Aggregation for Misinformation Detection](https://mlsanity.com/paper/2604.16172/) (2026)

## Abstract (arXiv)

The rise of multimodal misinformation on social platforms poses significant challenges for individuals and societies. Its increased credibility and broader impact compared to textual misinformation make detection complex, requiring robust reasoning across diverse media types and profound knowledge for accurate verification. The emergence of Large Vision Language Model (LVLM) offers a potential solution to this problem. Leveraging their proficiency in processing visual and textual information, LVLM demonstrates promising capabilities in recognizing complex information and exhibiting strong reasoning skills. In this paper, we first investigate the potential of LVLM on multimodal misinformation detection. We find that even though LVLM has a superior performance compared to LLMs, its profound reasoning may present limited power with a lack of evidence. Based on these observations, we propose LEMMA: LVLM-Enhanced Multimodal Misinformation Detection with External Knowledge Augmentation. LEMMA leverages LVLM intuition and reasoning capabilities while augmenting them with external knowledge to enhance the accuracy of misinformation detection. Our method improves the accuracy over the top baseline LVLM by 7% and 13% on Twitter and Fakeddit datasets respectively.