# Interpretable Detection of Out-of-Context Misinformation with Neural-Symbolic-Enhanced Large Multimodal Model

arXiv: https://arxiv.org/abs/2304.07633 · HTML: https://mlsanity.com/paper/2304.07633/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 17 of its 27 references that are in the map.
- First posted on arXiv 15 Apr 2023.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Fake News Detection on Social Media: A Data Mining Perspective](https://mlsanity.com/paper/1708.01967/) (2017)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [Visual News: Benchmark and Challenges in News Image Captioning](https://mlsanity.com/paper/2010.03743/) (2020)
- [NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media](https://mlsanity.com/paper/2104.05893/) (2021)
- [Identifying Coordinated Accounts on Social Media through Hidden Influence and Group Behaviours](https://mlsanity.com/paper/2008.11308/) (2020)
- [False Information on Web and Social Media: A Survey](https://mlsanity.com/paper/1804.08559/) (2018)
- [Multimedia Semantic Integrity Assessment Using Joint Embedding Of Images And Text](https://mlsanity.com/paper/1707.01606/) (2017)
- [Synthetic Misinformers: Generating and Combating Multimodal Misinformation](https://mlsanity.com/paper/2303.01217/) (2023)
- [Counterfactual Neural Temporal Point Process for Estimating Causal Influence of Misinformation on Social Media](https://mlsanity.com/paper/2210.07518/) (2022)
- [VigDet: Knowledge Informed Neural Temporal Point Process for Coordination Detection on Social Media](https://mlsanity.com/paper/2110.15454/) (2021)
- [COVID-19 Vaccine Misinformation Campaigns and Social Media Narratives](https://mlsanity.com/paper/2106.08423/) (2021)
- [Generalization Differences between End-to-End and Neuro-Symbolic Vision-Language Reasoning Systems](https://mlsanity.com/paper/2210.15037/) (2022)
- [COVID-19 on Social Media: Analyzing Misinformation in Twitter Conversations](https://mlsanity.com/paper/2003.12309/) (2020)
- [Construction of Large-Scale Misinformation Labeled Datasets from Social Media Discourse using Label Refinement](https://mlsanity.com/paper/2202.12413/) (2022)

## Built on by (most cited first, 8 of 8)

- [SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection](https://mlsanity.com/paper/2403.03170/) (2024)
- [Multimodal Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2305.13507/) (2023)
- [Leveraging Chat-Based Large Vision Language Models for Multimodal Out-Of-Context Detection](https://mlsanity.com/paper/2403.08776/) (2024)
- [MAD-Sherlock: Multi-Agent Debate for Visual Misinformation Detection](https://mlsanity.com/paper/2410.20140/) (2024)
- [EXCLAIM: An Explainable Cross-Modal Agentic System for Misinformation Detection with Hierarchical Retrieval](https://mlsanity.com/paper/2504.06269/) (2025)
- [Image-Text Out-Of-Context Detection Using Synthetic Multimodal Misinformation](https://mlsanity.com/paper/2403.08783/) (2024)
- [Augmented Vision-Language Models: A Systematic Review](https://mlsanity.com/paper/2507.22933/) (2025)
- [NepOOC-M: Bilingual Nepali-English Benchmark and Comparative Analysis of Multimodal Architectures for OOC Detection](https://mlsanity.com/paper/2608.19212/) (2026)

## Abstract (arXiv)

Recent years have witnessed the sustained evolution of misinformation that aims at manipulating public opinions. Unlike traditional rumors or fake news editors who mainly rely on generated and/or counterfeited images, text and videos, current misinformation creators now more tend to use out-of-context multimedia contents (e.g. mismatched images and captions) to deceive the public and fake news detection systems. This new type of misinformation increases the difficulty of not only detection but also clarification, because every individual modality is close enough to true information. To address this challenge, in this paper we explore how to achieve interpretable cross-modal de-contextualization detection that simultaneously identifies the mismatched pairs and the cross-modal contradictions, which is helpful for fact-check websites to document clarifications. The proposed model first symbolically disassembles the text-modality information to a set of fact queries based on the Abstract Meaning Representation of the caption and then forwards the query-image pairs into a pre-trained large vision-language model select the ``evidences" that are helpful for us to detect misinformation. Extensive experiments indicate that the proposed methodology can provide us with much more interpretable predictions while maintaining the accuracy same as the state-of-the-art model on this task.