# COSMOS: Catching Out-of-Context Misinformation with Self-Supervised Learning

arXiv: https://arxiv.org/abs/2101.06278 · HTML: https://mlsanity.com/paper/2101.06278/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 53 references that are in the map.
- First posted on arXiv 15 Jan 2021.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Face X-ray for More General Face Forgery Detection](https://mlsanity.com/paper/1912.13458/) (2019)
- [MesoNet: a Compact Facial Video Forgery Detection Network](https://mlsanity.com/paper/1809.00888/) (2018)
- ["Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection](https://mlsanity.com/paper/1705.00648/) (2017)
- [Exposing DeepFake Videos By Detecting Face Warping Artifacts](https://mlsanity.com/paper/1811.00656/) (2018)
- [Exposing Deep Fakes Using Inconsistent Head Poses](https://mlsanity.com/paper/1811.00661/) (2018)
- [Media Forensics and DeepFakes: an overview](https://mlsanity.com/paper/2001.06564/) (2020)
- [Capsule-Forensics: Using Capsule Networks to Detect Forged Images and Videos](https://mlsanity.com/paper/1810.11215/) (2018)
- [Generating Fact Checking Explanations](https://mlsanity.com/paper/2004.05773/) (2020)
- [Two-Stream Neural Networks for Tampered Face Detection](https://mlsanity.com/paper/1803.11276/) (2018)
- [ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection](https://mlsanity.com/paper/1812.02510/) (2018)
- [CSI: A Hybrid Deep Model for Fake News Detection](https://mlsanity.com/paper/1703.06959/) (2017)
- [ID-Reveal: Identity-aware DeepFake Video Detection](https://mlsanity.com/paper/2012.02512/) (2020)
- [Fighting an Infodemic: COVID-19 Fake News Dataset](https://mlsanity.com/paper/2011.03327/) (2020)
- [SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models](https://mlsanity.com/paper/2002.06652/) (2020)
- [Fact-Checking Meets Fauxtography: Verifying Claims About Images](https://mlsanity.com/paper/1908.11722/) (2019)
- [Detecting Cross-Modal Inconsistency to Defend Against Neural Fake News](https://mlsanity.com/paper/2009.07698/) (2020)
- [Multi-Hop Fact Checking of Political Claims](https://mlsanity.com/paper/2009.06401/) (2020)
- [EmbraceNet: A robust deep learning architecture for multimodal classification](https://mlsanity.com/paper/1904.09078/) (2019)
- [Generalized Zero and Few-Shot Transfer for Facial Forgery Detection](https://mlsanity.com/paper/2006.11863/) (2020)
- [Evaluating Deep Learning Approaches for Covid19 Fake News Detection](https://mlsanity.com/paper/2101.04012/) (2021)
- [It Takes Nine to Smell a Rat: Neural Multi-Task Learning for Check-Worthiness Prediction](https://mlsanity.com/paper/1908.07912/) (2019)
- [Automatic Fact-Checking Using Context and Discourse Information](https://mlsanity.com/paper/1908.01328/) (2019)

## Built on by (most cited first, 27 of 27)

- [NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media](https://mlsanity.com/paper/2104.05893/) (2021)
- [MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMs](https://mlsanity.com/paper/2406.08772/) (2024)
- [Detecting and Grounding Multi-Modal Media Manipulation and Beyond](https://mlsanity.com/paper/2309.14203/) (2023)
- [Multimodal Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2305.13507/) (2023)
- [VERITE: A Robust Benchmark for Multimodal Misinformation Detection Accounting for Unimodal Bias](https://mlsanity.com/paper/2304.14133/) (2023)
- [Synthetic Misinformers: Generating and Combating Multimodal Misinformation](https://mlsanity.com/paper/2303.01217/) (2023)
- [Matching Tweets With Applicable Fact-Checks Across Languages](https://mlsanity.com/paper/2202.07094/) (2022)
- [Cheap-fake Detection with LLM using Prompt Engineering](https://mlsanity.com/paper/2306.02776/) (2023)
- [MFC-Bench: Benchmarking Multimodal Fact-Checking with Large Vision-Language Models](https://mlsanity.com/paper/2406.11288/) (2024)
- [Multi-Modal Semantic Inconsistency Detection in Social Media News Posts](https://mlsanity.com/paper/2105.12855/) (2021)
- [Credible, Unreliable or Leaked?: Evidence Verification for Enhanced Automated Fact-checking](https://mlsanity.com/paper/2404.18971/) (2024)
- [Harnessing the Power of Text-image Contrastive Models for Automatic Detection of Online Misinformation](https://mlsanity.com/paper/2304.10249/) (2023)
- [CMIE: Combining MLLM Insights with External Evidence for Explainable Out-of-Context Misinformation Detection](https://mlsanity.com/paper/2505.23449/) (2025)
- [The Paradigm Shift: A Comprehensive Survey on Large Vision Language Models for Multimodal Fake News Detection](https://mlsanity.com/paper/2601.15316/) (2026)
- [NewsStories: Illustrating articles with visual summaries](https://mlsanity.com/paper/2207.13061/) (2022)
- [T^2Agent A Tool-augmented Multimodal Misinformation Detection Agent with Monte Carlo Tree Search](https://mlsanity.com/paper/2505.19768/) (2025)
- [Detecting Out-of-Context Image-Caption Pairs in News: A Counter-Intuitive Method](https://mlsanity.com/paper/2308.16611/) (2023)
- [Image-Text Out-Of-Context Detection Using Synthetic Multimodal Misinformation](https://mlsanity.com/paper/2403.08783/) (2024)
- [Interpretable Multimodal Out-of-context Detection with Soft Logic Regularization](https://mlsanity.com/paper/2406.04756/) (2024)
- [When Seeing Is Not Believing -- A Benchmark for Search-Grounded Video Misinformation Detection](https://mlsanity.com/paper/2606.04098/) (2026)
- [Grand Challenge On Detecting Cheapfakes](https://mlsanity.com/paper/2304.01328/) (2023)
- [Fact or Fake? Assessing the Role of Deepfake Detectors in Multimodal Misinformation Detection](https://mlsanity.com/paper/2602.01854/) (2026)
- [PIXELMOD: Improving Soft Moderation of Visual Misleading Information on Twitter](https://mlsanity.com/paper/2407.20987/) (2024)
- [Towards Explainable Bilingual Multimodal Misinformation Detection and Localization](https://mlsanity.com/paper/2506.22930/) (2025)
- [D-SECURE: Dual-Source Evidence Combination for Unified Reasoning in Misinformation Detection](https://mlsanity.com/paper/2602.14441/) (2026)
- [T-IMPACT: A Severity-Aware Benchmark for Contextual Image-Text Manipulation](https://mlsanity.com/paper/2606.22339/) (2026)
- [Verification-Notebook Learning for Source-Aware Multimodal Misinformation Detection](https://mlsanity.com/paper/2607.23581/) (2026)

## Abstract (arXiv)

Despite the recent attention to DeepFakes, one of the most prevalent ways to mislead audiences on social media is the use of unaltered images in a new but false context. To address these challenges and support fact-checkers, we propose a new method that automatically detects out-of-context image and text pairs. Our key insight is to leverage the grounding of image with text to distinguish out-of-context scenarios that cannot be disambiguated with language alone. We propose a self-supervised training strategy where we only need a set of captioned images. At train time, our method learns to selectively align individual objects in an image with textual claims, without explicit supervision. At test time, we check if both captions correspond to the same object(s) in the image but are semantically different, which allows us to make fairly accurate out-of-context predictions. Our method achieves 85% out-of-context detection accuracy. To facilitate benchmarking of this task, we create a large-scale dataset of 200K images with 450K textual captions from a variety of news websites, blogs, and social media posts. The dataset and source code is publicly available at https://shivangi-aneja.github.io/projects/cosmos/.