# Interpretable Multimodal Out-of-context Detection with Soft Logic Regularization

arXiv: https://arxiv.org/abs/2406.04756 · HTML: https://mlsanity.com/paper/2406.04756/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 32 references that are in the map.
- First posted on arXiv 7 Jun 2024.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Neural Probabilistic Logic Programming in DeepProbLog](https://mlsanity.com/paper/1907.08194/) (2019)
- [Visual News: Benchmark and Challenges in News Image Captioning](https://mlsanity.com/paper/2010.03743/) (2020)
- [NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media](https://mlsanity.com/paper/2104.05893/) (2021)
- [Graph Neural Networks Meet Neural-Symbolic Computing: A Survey and Perspective](https://mlsanity.com/paper/2003.00330/) (2020)
- [Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resources](https://mlsanity.com/paper/2112.00061/) (2021)
- [Probabilistic Logic Neural Networks for Reasoning](https://mlsanity.com/paper/1906.08495/) (2019)
- [A Logic-Driven Framework for Consistency of Neural Models](https://mlsanity.com/paper/1909.00126/) (2019)
- [Uncertain Natural Language Inference](https://mlsanity.com/paper/1909.03042/) (2019)
- [Towards Interpretable Natural Language Understanding with Explanations as Latent Variables](https://mlsanity.com/paper/2011.05268/) (2020)
- [Evidence-aware Fake News Detection with Graph Neural Networks](https://mlsanity.com/paper/2201.06885/) (2022)
- [COSMOS: Catching Out-of-Context Misinformation with Self-Supervised Learning](https://mlsanity.com/paper/2101.06278/) (2021)
- [LOREN: Logic-Regularized Reasoning for Interpretable Fact Verification](https://mlsanity.com/paper/2012.13577/) (2020)
- [Integrating Deep Learning with Logic Fusion for Information Extraction](https://mlsanity.com/paper/1912.03041/) (2019)
- [Multi-Modal Semantic Inconsistency Detection in Social Media News Posts](https://mlsanity.com/paper/2105.12855/) (2021)
- [Multi-View Graph Representation Learning for Answering Hybrid Numerical Reasoning Question](https://mlsanity.com/paper/2305.03458/) (2023)
- [Lifted Relational Neural Networks](https://mlsanity.com/paper/1508.05128/) (2015)

## Built on by (most cited first, 2 of 2)

- [EXCLAIM: An Explainable Cross-Modal Agentic System for Misinformation Detection with Hierarchical Retrieval](https://mlsanity.com/paper/2504.06269/) (2025)
- [Multi-MLLM Knowledge Distillation for Out-of-Context News Detection](https://mlsanity.com/paper/2505.22517/) (2025)

## Abstract (arXiv)

The rapid spread of information through mobile devices and media has led to the widespread of false or deceptive news, causing significant concerns in society. Among different types of misinformation, image repurposing, also known as out-of-context misinformation, remains highly prevalent and effective. However, current approaches for detecting out-of-context misinformation often lack interpretability and offer limited explanations. In this study, we propose a logic regularization approach for out-of-context detection called LOGRAN (LOGic Regularization for out-of-context ANalysis). The primary objective of LOGRAN is to decompose the out-of-context detection at the phrase level. By employing latent variables for phrase-level predictions, the final prediction of the image-caption pair can be aggregated using logical rules. The latent variables also provide an explanation for how the final result is derived, making this fine-grained detection method inherently explanatory. We evaluate the performance of LOGRAN on the NewsCLIPpings dataset, showcasing competitive overall results. Visualized examples also reveal faithful phrase-level predictions of out-of-context images, accompanied by explanations. This highlights the effectiveness of our approach in addressing out-of-context detection and enhancing interpretability.