# Multimodal Analytics for Real-world News using Measures of Cross-modal Entity Consistency

arXiv: https://arxiv.org/abs/2003.10421 · HTML: https://mlsanity.com/paper/2003.10421/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Mar 2020.

## Built on by (most cited first, 30 of 33)

- [NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media](https://mlsanity.com/paper/2104.05893/) (2021)
- [SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection](https://mlsanity.com/paper/2403.03170/) (2024)
- [Multimodal Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2305.13507/) (2023)
- [A Survey on Multimodal Disinformation Detection](https://mlsanity.com/paper/2103.12541/) (2021)
- [DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts](https://mlsanity.com/paper/2412.10510/) (2024)
- [VERITE: A Robust Benchmark for Multimodal Misinformation Detection Accounting for Unimodal Bias](https://mlsanity.com/paper/2304.14133/) (2023)
- [Synthetic Misinformers: Generating and Combating Multimodal Misinformation](https://mlsanity.com/paper/2303.01217/) (2023)
- [Support or Refute: Analyzing the Stance of Evidence to Detect Out-of-Context Mis- and Disinformation](https://mlsanity.com/paper/2311.01766/) (2023)
- [RED-DOT: Multimodal Fact-checking via Relevant Evidence Detection](https://mlsanity.com/paper/2311.09939/) (2023)
- [Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation](https://mlsanity.com/paper/2112.08594/) (2021)
- [Multi-modal Fake News Detection on Social Media via Multi-grained Information Fusion](https://mlsanity.com/paper/2304.00827/) (2023)
- [COVE: COntext and VEracity prediction for out-of-context images](https://mlsanity.com/paper/2502.01194/) (2025)
- [Multimodal Misinformation Detection by Learning from Synthetic Data with Multimodal LLMs](https://mlsanity.com/paper/2409.19656/) (2024)
- [Detecting Misinformation in Multimedia Content through Cross-Modal Entity Consistency: A Dual Learning Approach](https://mlsanity.com/paper/2409.00022/) (2024)
- [Similarity over Factuality: Are we making progress on multimodal out-of-context misinformation detection?](https://mlsanity.com/paper/2407.13488/) (2024)
- [M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset](https://mlsanity.com/paper/2510.23508/) (2025)
- [CMIE: Combining MLLM Insights with External Evidence for Explainable Out-of-Context Misinformation Detection](https://mlsanity.com/paper/2505.23449/) (2025)
- [Verifying Cross-modal Entity Consistency in News using Vision-language Models](https://mlsanity.com/paper/2501.11403/) (2025)
- ["Image, Tell me your story!" Predicting the original meta-context of visual misinformation](https://mlsanity.com/paper/2408.09939/) (2024)
- [MONet: Multi-scale Overlap Network for Duplication Detection in Biomedical Images](https://mlsanity.com/paper/2207.09107/) (2022)
- [Assessing News Thumbnail Representativeness: Counterfactual text can enhance the cross-modal matching ability](https://mlsanity.com/paper/2402.11159/) (2024)
- [Image-Text Out-Of-Context Detection Using Synthetic Multimodal Misinformation](https://mlsanity.com/paper/2403.08783/) (2024)
- [The Synthetic Media Shift: Tracking the Rise, Virality, and Detectability of AI-Generated Multimodal Misinformation](https://mlsanity.com/paper/2604.15372/) (2026)
- [RW-Post: Auditable Evidence-Grounded Multimodal Fact-Checking in the Wild](https://mlsanity.com/paper/2605.10357/) (2026)
- [Mitigating GenAI-Powered Evidence Pollution for Out-Of-Context Misinformation Detection](https://mlsanity.com/paper/2501.14728/) (2025)
- [Latent Reconstruction from Generated Data for Multimodal Misinformation Detection](https://mlsanity.com/paper/2504.06010/) (2025)
- [Out-of-Context Misinformation Detection via Variational Domain-Invariant Learning with Test-Time Training](https://mlsanity.com/paper/2511.10213/) (2025)
- [Exposing Cross-Modal Consistency for Fake News Detection in Short-Form Videos](https://mlsanity.com/paper/2603.14992/) (2026)
- [A Review of Web Infodemic Analysis and Detection Trends across Multi-modalities using Deep Neural Networks](https://mlsanity.com/paper/2112.00803/) (2021)
- [SceneGraMMi: Scene Graph-boosted Hybrid-fusion for Multi-Modal Misinformation Veracity Prediction](https://mlsanity.com/paper/2410.15517/) (2024)

## Abstract (arXiv)

The World Wide Web has become a popular source for gathering information and news. Multimodal information, e.g., enriching text with photos, is typically used to convey the news more effectively or to attract attention. Photo content can range from decorative, depict additional important information, or can even contain misleading information. Therefore, automatic approaches to quantify cross-modal consistency of entity representation can support human assessors to evaluate the overall multimodal message, for instance, with regard to bias or sentiment. In some cases such measures could give hints to detect fake news, which is an increasingly important topic in today's society. In this paper, we introduce a novel task of cross-modal consistency verification in real-world news and present a multimodal approach to quantify the entity coherence between image and text. Named entity linking is applied to extract persons, locations, and events from news texts. Several measures are suggested to calculate cross-modal similarity for these entities using state of the art approaches. In contrast to previous work, our system automatically gathers example data from the Web and is applicable to real-world news. Results on two novel datasets that cover different languages, topics, and domains demonstrate the feasibility of our approach. Datasets and code are publicly available to foster research towards this new direction.