# Harnessing the Power of Text-image Contrastive Models for Automatic Detection of Online Misinformation

arXiv: https://arxiv.org/abs/2304.10249 · HTML: https://mlsanity.com/paper/2304.10249/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 19 Apr 2023.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Media](https://mlsanity.com/paper/1809.01286/) (2018)
- [Exposing GAN-generated Faces Using Inconsistent Corneal Specular Highlights](https://mlsanity.com/paper/2009.11924/) (2020)
- [SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models](https://mlsanity.com/paper/2002.06652/) (2020)
- [Eyes Tell All: Irregular Pupil Shapes Reveal GAN-generated Faces](https://mlsanity.com/paper/2109.00162/) (2021)
- [Generalizing to the Future: Mitigating Entity Bias in Fake News Detection](https://mlsanity.com/paper/2204.09484/) (2022)
- [GAN-generated Faces Detection: A Survey and New Perspectives](https://mlsanity.com/paper/2202.07145/) (2022)
- [The Surprising Performance of Simple Baselines for Misinformation Detection](https://mlsanity.com/paper/2104.06952/) (2021)
- [Robust Attentive Deep Neural Network for Exposing GAN-generated Faces](https://mlsanity.com/paper/2109.02167/) (2021)
- [COSMOS: Catching Out-of-Context Misinformation with Self-Supervised Learning](https://mlsanity.com/paper/2101.06278/) (2021)
- [Rank-based Decomposable Losses in Machine Learning: A Survey](https://mlsanity.com/paper/2207.08768/) (2022)
- [Evaluating Deep Learning Approaches for Covid19 Fake News Detection](https://mlsanity.com/paper/2101.04012/) (2021)
- [Open-Eye: An Open Platform to Study Human Performance on Identifying AI-Synthesized Faces](https://mlsanity.com/paper/2205.06680/) (2022)
- [The Future of Misinformation Detection: New Perspectives and Trends](https://mlsanity.com/paper/1909.03654/) (2019)
- [PseudoProp: Robust Pseudo-Label Generation for Semi-Supervised Object Detection in Autonomous Driving Systems](https://mlsanity.com/paper/2203.05983/) (2022)
- [ACM Multimedia Grand Challenge on Detecting Cheapfakes](https://mlsanity.com/paper/2207.14534/) (2022)
- [Determining the Veracity of Rumours on Twitter](https://mlsanity.com/paper/1611.06314/) (2016)

## Built on by (most cited first, 6 of 6)

- [Preserving Fairness Generalization in Deepfake Detection](https://mlsanity.com/paper/2402.17229/) (2024)
- [Improving Generalization of Medical Image Registration Foundation Model](https://mlsanity.com/paper/2505.06527/) (2025)
- [Robust AI-Generated Face Detection with Imbalanced Data](https://mlsanity.com/paper/2505.02182/) (2025)
- [A Self-Learning Multimodal Approach for Fake News Detection](https://mlsanity.com/paper/2412.05843/) (2024)
- [Revisiting Deepfake Detection: Chronological Continual Learning and the Limits of Generalization](https://mlsanity.com/paper/2509.07993/) (2025)
- [SynthGuard: An Open Platform for Detecting AI-Generated Multimedia with Multimodal LLMs](https://mlsanity.com/paper/2511.12404/) (2025)

## Abstract (arXiv)

As growing usage of social media websites in the recent decades, the amount of news articles spreading online rapidly, resulting in an unprecedented scale of potentially fraudulent information. Although a plenty of studies have applied the supervised machine learning approaches to detect such content, the lack of gold standard training data has hindered the development. Analysing the single data format, either fake text description or fake image, is the mainstream direction for the current research. However, the misinformation in real-world scenario is commonly formed as a text-image pair where the news article/news title is described as text content, and usually followed by the related image. Given the strong ability of learning features without labelled data, contrastive learning, as a self-learning approach, has emerged and achieved success on the computer vision. In this paper, our goal is to explore the constrastive learning in the domain of misinformation identification. We developed a self-learning model and carried out the comprehensive experiments on a public data set named COSMOS. Comparing to the baseline classifier, our model shows the superior performance of non-matched image-text pair detection (approximately 10%) when the training data is insufficient. In addition, we observed the stability for contrsative learning and suggested the use of it offers large reductions in the number of training data, whilst maintaining comparable classification results.