# Multimodal Fake News Detection via CLIP-Guided Learning

arXiv: https://arxiv.org/abs/2205.14304 · HTML: https://mlsanity.com/paper/2205.14304/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 49 references that are in the map.
- First posted on arXiv 28 May 2022.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Media](https://mlsanity.com/paper/1809.01286/) (2018)
- [Rumor Detection on Social Media with Bi-Directional Graph Convolutional Networks](https://mlsanity.com/paper/2001.06362/) (2020)
- [Image Manipulation Detection by Multi-View Multi-Scale Supervision](https://mlsanity.com/paper/2104.06832/) (2021)
- [Detection and Resolution of Rumours in Social Media: A Survey](https://mlsanity.com/paper/1704.00656/) (2017)
- [A Stylometric Inquiry into Hyperpartisan and Fake News](https://mlsanity.com/paper/1702.05638/) (2017)
- [HairCLIP: Design Your Hair by Text and Reference Image](https://mlsanity.com/paper/2112.05142/) (2021)
- [Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering](https://mlsanity.com/paper/2104.03149/) (2021)
- [CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classification](https://mlsanity.com/paper/2204.14244/) (2022)
- [Improving Fake News Detection by Using an Entity-enhanced Framework to Fuse Diverse Multimodal Clues](https://mlsanity.com/paper/2108.10509/) (2021)
- [Graph Neural Networks with Continual Learning for Fake News Detection from Social Media](https://mlsanity.com/paper/2007.03316/) (2020)
- [Leveraging Multi-Source Weak Social Supervision for Early Detection of Fake News](https://mlsanity.com/paper/2004.01732/) (2020)
- [Explainable Tsetlin Machine framework for fake news detection with credibility score assessment](https://mlsanity.com/paper/2105.09114/) (2021)

## Built on by (most cited first, 30 of 42)

- [VERITE: A Robust Benchmark for Multimodal Misinformation Detection Accounting for Unimodal Bias](https://mlsanity.com/paper/2304.14133/) (2023)
- [Evidence-Driven Retrieval Augmented Response Generation for Online Misinformation](https://mlsanity.com/paper/2403.14952/) (2024)
- [MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection System](https://mlsanity.com/paper/2307.07135/) (2023)
- [Multi-modal Machine Learning in Engineering Design: A Review and Future Directions](https://mlsanity.com/paper/2302.10909/) (2023)
- [Retrieval Augmented Fact Verification by Synthesizing Contrastive Arguments](https://mlsanity.com/paper/2406.09815/) (2024)
- [Cross-modal Contrastive Learning for Multimodal Fake News Detection](https://mlsanity.com/paper/2302.14057/) (2023)
- [Multi-modal Fake News Detection on Social Media via Multi-grained Information Fusion](https://mlsanity.com/paper/2304.00827/) (2023)
- [GAMED: Knowledge Adaptive Multi-Experts Decoupling for Multimodal Fake News Detection](https://mlsanity.com/paper/2412.12164/) (2024)
- [Modality Interactive Mixture-of-Experts for Fake News Detection](https://mlsanity.com/paper/2501.12431/) (2025)
- [Semantic Shield: Defending Vision-Language Models Against Backdooring and Poisoning via Fine-grained Knowledge Alignment](https://mlsanity.com/paper/2411.15673/) (2024)
- [The Paradigm Shift: A Comprehensive Survey on Large Vision Language Models for Multimodal Fake News Detection](https://mlsanity.com/paper/2601.15316/) (2026)
- [Similarity-Aware Multimodal Prompt Learning for Fake News Detection](https://mlsanity.com/paper/2304.04187/) (2023)
- [MMCFND: Multimodal Multilingual Caption-aware Fake News Detection for Low-resource Indic Languages](https://mlsanity.com/paper/2410.10407/) (2024)
- [RAMA: Retrieval-Augmented Multi-Agent Framework for Misinformation Detection in Multimodal Fact-Checking](https://mlsanity.com/paper/2507.09174/) (2025)
- [Semantics-Oriented Multitask Learning for DeepFake Detection: A Joint Embedding Approach](https://mlsanity.com/paper/2408.16305/) (2024)
- [KEN: Knowledge Augmentation and Emotion Guidance Network for Multimodal Fake News Detection](https://mlsanity.com/paper/2507.09647/) (2025)
- [TT-BLIP: Enhancing Fake News Detection Using BLIP and Tri-Transformer](https://mlsanity.com/paper/2403.12481/) (2024)
- [Search, Examine and Early-Termination: Fake News Detection with Annotation-Free Evidences](https://mlsanity.com/paper/2407.07931/) (2024)
- [Cross-Modal Augmentation for Few-Shot Multimodal Fake News Detection](https://mlsanity.com/paper/2407.12880/) (2024)
- [CroMe: Multimodal Fake News Detection using Cross-Modal Tri-Transformer and Metric Learning](https://mlsanity.com/paper/2501.12422/) (2025)
- [SEER: Semantic Enhancement and Emotional Reasoning Network for Multimodal Fake News Detection](https://mlsanity.com/paper/2507.13415/) (2025)
- [Fine-grained Multiple Supervisory Network for Multi-modal Manipulation Detecting and Grounding](https://mlsanity.com/paper/2508.02479/) (2025)
- [Knowledge-Enhanced Hierarchical Information Correlation Learning for Multi-Modal Rumor Detection](https://mlsanity.com/paper/2306.15946/) (2023)
- [Can Out-of-Domain data help to Learn Domain-Specific Prompts for Multimodal Misinformation Detection?](https://mlsanity.com/paper/2311.16496/) (2023)
- [UNITE-FND: Reframing Multimodal Fake News Detection through Unimodal Scene Translation](https://mlsanity.com/paper/2502.11132/) (2025)
- [Effectiveness of Large Multimodal Models in Detecting Disinformation: Experimental Results](https://mlsanity.com/paper/2509.22377/) (2025)
- [Disentangling Fact from Sentiment: A Dynamic Conflict-Consensus Framework for Multimodal Fake News Detection](https://mlsanity.com/paper/2512.20670/) (2025)
- [Multimodal Misinformation Detection in a South African Social Media Environment](https://mlsanity.com/paper/2312.04052/) (2023)
- [Fact-checking based fake news detection: a review](https://mlsanity.com/paper/2401.01717/) (2024)
- [Multi-method Integration with Confidence-based Weighting for Zero-shot Image Classification](https://mlsanity.com/paper/2405.02155/) (2024)

## Abstract (arXiv)

Multimodal fake news detection has attracted many research interests in social forensics. Many existing approaches introduce tailored attention mechanisms to guide the fusion of unimodal features. However, how the similarity of these features is calculated and how it will affect the decision-making process in FND are still open questions. Besides, the potential of pretrained multi-modal feature learning models in fake news detection has not been well exploited. This paper proposes a FND-CLIP framework, i.e., a multimodal Fake News Detection network based on Contrastive Language-Image Pretraining (CLIP). Given a targeted multimodal news, we extract the deep representations from the image and text using a ResNet-based encoder, a BERT-based encoder and two pair-wise CLIP encoders. The multimodal feature is a concatenation of the CLIP-generated features weighted by the standardized cross-modal similarity of the two modalities. The extracted features are further processed for redundancy reduction before feeding them into the final classifier. We introduce a modality-wise attention module to adaptively reweight and aggregate the features. We have conducted extensive experiments on typical fake news datasets. The results indicate that the proposed framework has a better capability in mining crucial features for fake news detection. The proposed FND-CLIP can achieve better performances than previous works, i.e., 0.7%, 6.8% and 1.3% improvements in overall accuracy on Weibo, Politifact and Gossipcop, respectively. Besides, we justify that CLIP-based learning can allow better flexibility on multimodal feature selection.