# Detecting Misinformation in Multimedia Content through Cross-Modal Entity Consistency: A Dual Learning Approach

arXiv: https://arxiv.org/abs/2409.00022 · HTML: https://mlsanity.com/paper/2409.00022/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 66 references that are in the map.
- First posted on arXiv 16 Aug 2024.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Fake News Detection on Social Media: A Data Mining Perspective](https://mlsanity.com/paper/1708.01967/) (2017)
- [Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge](https://mlsanity.com/paper/1609.06647/) (2016)
- [Vision Transformers are Parameter-Efficient Audio-Visual Learners](https://mlsanity.com/paper/2212.07983/) (2022)
- [Improving Fake News Detection by Using an Entity-enhanced Framework to Fuse Diverse Multimodal Clues](https://mlsanity.com/paper/2108.10509/) (2021)
- [Detecting Cross-Modal Inconsistency to Defend Against Neural Fake News](https://mlsanity.com/paper/2009.07698/) (2020)
- [FakeSV: A Multimodal Benchmark with Rich Social Context for Fake News Detection on Short Video Platforms](https://mlsanity.com/paper/2211.10973/) (2022)
- [Multimodal Variational Auto-encoder based Audio-Visual Segmentation](https://mlsanity.com/paper/2310.08303/) (2023)
- [Multimodal Analytics for Real-world News using Measures of Cross-modal Entity Consistency](https://mlsanity.com/paper/2003.10421/) (2020)
- [Cross-modal Contrastive Learning for Multimodal Fake News Detection](https://mlsanity.com/paper/2302.14057/) (2023)
- [Towards Automatic Detection of Misinformation in Online Medical Videos](https://mlsanity.com/paper/1909.01543/) (2019)
- [Multimodal Emergent Fake News Detection via Meta Neural Process Networks](https://mlsanity.com/paper/2106.13711/) (2021)
- [Semi-supervised Content-based Detection of Misinformation via Tensor Embeddings](https://mlsanity.com/paper/1804.09088/) (2018)
- [Combating Online Misinformation Videos: Characterization, Detection, and Future Directions](https://mlsanity.com/paper/2302.03242/) (2023)
- [Inconsistent Matters: A Knowledge-guided Dual-consistency Network for Multi-modal Rumor Detection](https://mlsanity.com/paper/2306.02137/) (2023)
- [Misleading Metadata Detection on YouTube](https://mlsanity.com/paper/1901.08759/) (2019)
- [Detecting Medical Misinformation on Social Media Using Multimodal Deep Learning](https://mlsanity.com/paper/2012.13968/) (2020)
- [Investigating Online Financial Misinformation and Its Consequences: A Computational Perspective](https://mlsanity.com/paper/2309.12363/) (2023)
- [Misinformation Detection in Social Media Video Posts](https://mlsanity.com/paper/2202.07706/) (2022)
- [Image Credibility Analysis with Effective Domain Transferred Deep Networks](https://mlsanity.com/paper/1611.05328/) (2016)

## Built on by (most cited first, 12 of 12)

- [DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts](https://mlsanity.com/paper/2412.10510/) (2024)
- [Comprehensive Evaluation of Multimodal AI Models in Medical Imaging Diagnosis: From Data Augmentation to Preference-Based Comparison](https://mlsanity.com/paper/2412.05536/) (2024)
- [Challenges and Innovations in LLM-Powered Fake News Detection: A Synthesis of Approaches and Future Directions](https://mlsanity.com/paper/2502.00339/) (2025)
- [Journey into Automation: Image-Derived Pavement Texture Extraction and Evaluation](https://mlsanity.com/paper/2501.02414/) (2025)
- [Comprehensive Overview of Artificial Intelligence Applications in Modern Industries](https://mlsanity.com/paper/2409.13059/) (2024)
- [T^2Agent A Tool-augmented Multimodal Misinformation Detection Agent with Monte Carlo Tree Search](https://mlsanity.com/paper/2505.19768/) (2025)
- [DyConfidMatch: Dynamic Thresholding and Re-sampling for 3D Semi-supervised Learning](https://mlsanity.com/paper/2411.08340/) (2024)
- [ChromouVQA: Benchmarking Vision-Language Models under Chromatic Camouflaged Images](https://mlsanity.com/paper/2512.05137/) (2025)
- [Pre-trained Graphformer-based Ranking at Web-scale Search (Extended Abstract)](https://mlsanity.com/paper/2409.16590/) (2024)
- [Generative Pre-trained Ranking Model with Over-parameterization at Web-Scale (Extended Abstract)](https://mlsanity.com/paper/2409.16594/) (2024)
- [CROPS: A Deployable Crop Management System Over All Possible State Availabilities](https://mlsanity.com/paper/2411.06034/) (2024)
- [MMTM: Tri-Modal Topic Modeling for Long-Form Video via Similarity-Gated Fusion](https://mlsanity.com/paper/2605.29765/) (2026)

## Abstract (arXiv)

The landscape of social media content has evolved significantly, extending from text to multimodal formats. This evolution presents a significant challenge in combating misinformation. Previous research has primarily focused on single modalities or text-image combinations, leaving a gap in detecting multimodal misinformation. While the concept of entity consistency holds promise in detecting multimodal misinformation, simplifying the representation to a scalar value overlooks the inherent complexities of high-dimensional representations across different modalities. To address these limitations, we propose a Multimedia Misinformation Detection (MultiMD) framework for detecting misinformation from video content by leveraging cross-modal entity consistency. The proposed dual learning approach allows for not only enhancing misinformation detection performance but also improving representation learning of entity consistency across different modalities. Our results demonstrate that MultiMD outperforms state-of-the-art baseline models and underscore the importance of each modality in misinformation detection. Our research provides novel methodological and technical insights into multimodal misinformation detection.