# Cross-modal Contrastive Learning for Multimodal Fake News Detection

arXiv: https://arxiv.org/abs/2302.14057 · HTML: https://mlsanity.com/paper/2302.14057/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 54 references that are in the map.
- First posted on arXiv 25 Feb 2023.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts](https://mlsanity.com/paper/2111.02358/) (2021)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [Detection and Resolution of Rumours in Social Media: A Survey](https://mlsanity.com/paper/1704.00656/) (2017)
- [PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining](https://mlsanity.com/paper/2204.14095/) (2022)
- [WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training](https://mlsanity.com/paper/2103.06561/) (2021)
- [A Stylometric Inquiry into Hyperpartisan and Fake News](https://mlsanity.com/paper/1702.05638/) (2017)
- [Exploiting Multi-domain Visual Information for Fake News Detection](https://mlsanity.com/paper/1908.04472/) (2019)
- [Multimodal Fake News Detection via CLIP-Guided Learning](https://mlsanity.com/paper/2205.14304/) (2022)
- [Explainable Tsetlin Machine framework for fake news detection with credibility score assessment](https://mlsanity.com/paper/2105.09114/) (2021)

## Built on by (most cited first, 23 of 23)

- [Detecting Misinformation in Multimedia Content through Cross-Modal Entity Consistency: A Dual Learning Approach](https://mlsanity.com/paper/2409.00022/) (2024)
- [Modality Interactive Mixture-of-Experts for Fake News Detection](https://mlsanity.com/paper/2501.12431/) (2025)
- [Semantic Shield: Defending Vision-Language Models Against Backdooring and Poisoning via Fine-grained Knowledge Alignment](https://mlsanity.com/paper/2411.15673/) (2024)
- [CMIE: Combining MLLM Insights with External Evidence for Explainable Out-of-Context Misinformation Detection](https://mlsanity.com/paper/2505.23449/) (2025)
- [FactGuard: Agentic Video Misinformation Detection via Reinforcement Learning](https://mlsanity.com/paper/2602.22963/) (2026)
- [Fake News Detection: It's All in the Data!](https://mlsanity.com/paper/2407.02122/) (2024)
- [Towards Unified Multimodal Misinformation Detection in Social Media: A Benchmark Dataset and Baseline](https://mlsanity.com/paper/2509.25991/) (2025)
- [Cross-Modal Augmentation for Few-Shot Multimodal Fake News Detection](https://mlsanity.com/paper/2407.12880/) (2024)
- [Large Visual-Language Models Are Also Good Classifiers: A Study of In-Context Multimodal Fake News Detection](https://mlsanity.com/paper/2407.12879/) (2024)
- [Consistent and Invariant Generalization Learning for Short-video Misinformation Detection](https://mlsanity.com/paper/2507.04061/) (2025)
- [A parameter-efficient three-branch architecture for multimodal misinformation detection with limited annotations](https://mlsanity.com/paper/2512.20257/) (2025)
- [Agentic Multi-Persona Framework for Evidence-Aware Fake News Detection](https://mlsanity.com/paper/2512.21039/) (2025)
- [Dynamic Analysis and Adaptive Discriminator for Fake News Detection](https://mlsanity.com/paper/2408.10883/) (2024)
- [MTPareto: A MultiModal Targeted Pareto Framework for Fake News Detection](https://mlsanity.com/paper/2501.06764/) (2025)
- [A Split-Window Transformer for Multi-Model Sequence Spammer Detection using Multi-Model Variational Autoencoder](https://mlsanity.com/paper/2502.16483/) (2025)
- [KGAlign: Joint Semantic-Structural Knowledge Encoding for Multimodal Fake News Detection](https://mlsanity.com/paper/2505.14714/) (2025)
- [ContextGuard-LVLM: Enhancing News Veracity through Fine-grained Cross-modal Contextual Consistency Verification](https://mlsanity.com/paper/2508.06623/) (2025)
- [Amplifying Prominent Representations in Multimodal Learning via Variational Dirichlet Process](https://mlsanity.com/paper/2510.20736/) (2025)
- [M4-BLIP: Advancing Multi-Modal Media Manipulation Detection through Face-Enhanced Local Analysis](https://mlsanity.com/paper/2512.01214/) (2025)
- [Multimodal rumor detection enhanced by external evidence and forgery features](https://mlsanity.com/paper/2601.14954/) (2026)
- [Harmful Visual Content Manipulation Matters in Misinformation Detection Under Multimedia Scenarios](https://mlsanity.com/paper/2603.21054/) (2026)
- [Bridging Pixels and Words: Mask-Aware Local Semantic Fusion for Multimodal Media Verification](https://mlsanity.com/paper/2603.26052/) (2026)
- [IDO: Incongruity-aware Distribution Optimization for Multimodal Fake News Detection](https://mlsanity.com/paper/2606.03418/) (2026)

## Abstract (arXiv)

Automatic detection of multimodal fake news has gained a widespread attention recently. Many existing approaches seek to fuse unimodal features to produce multimodal news representations. However, the potential of powerful cross-modal contrastive learning methods for fake news detection has not been well exploited. Besides, how to aggregate features from different modalities to boost the performance of the decision-making process is still an open question. To address that, we propose COOLANT, a cross-modal contrastive learning framework for multimodal fake news detection, aiming to achieve more accurate image-text alignment. To further improve the alignment precision, we leverage an auxiliary task to soften the loss term of negative samples during the contrast process. A cross-modal fusion module is developed to learn the cross-modality correlations. An attention mechanism with an attention guidance module is implemented to help effectively and interpretably aggregate the aligned unimodal representations and the cross-modality correlations. Finally, we evaluate the COOLANT and conduct a comparative study on two widely used datasets, Twitter and Weibo. The experimental results demonstrate that our COOLANT outperforms previous approaches by a large margin and achieves new state-of-the-art results on the two datasets.