# Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation

arXiv: https://arxiv.org/abs/2112.08594 · HTML: https://mlsanity.com/paper/2112.08594/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 29 references that are in the map.
- First posted on arXiv 16 Dec 2021.

## Builds on (most cited first, 12 of 12)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications](https://mlsanity.com/paper/2108.02818/) (2021)
- [NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media](https://mlsanity.com/paper/2104.05893/) (2021)
- [Accelerated Hierarchical Density Clustering](https://mlsanity.com/paper/1705.07321/) (2017)
- [Fighting an Infodemic: COVID-19 Fake News Dataset](https://mlsanity.com/paper/2011.03327/) (2020)
- [Multimodal Analytics for Real-world News using Measures of Cross-modal Entity Consistency](https://mlsanity.com/paper/2003.10421/) (2020)
- [A large-scale COVID-19 Twitter chatter dataset for open scientific research -- an international collaboration](https://mlsanity.com/paper/2004.03688/) (2020)
- [Multimedia Semantic Integrity Assessment Using Joint Embedding Of Images And Text](https://mlsanity.com/paper/1707.01606/) (2017)

## Built on by (most cited first, 16 of 16)

- [Multimodal Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2305.13507/) (2023)
- [DELL: Generating Reactions and Explanations for LLM-Based Misinformation Detection](https://mlsanity.com/paper/2402.10426/) (2024)
- [VERITE: A Robust Benchmark for Multimodal Misinformation Detection Accounting for Unimodal Bias](https://mlsanity.com/paper/2304.14133/) (2023)
- [Synthetic Misinformers: Generating and Combating Multimodal Misinformation](https://mlsanity.com/paper/2303.01217/) (2023)
- [RED-DOT: Multimodal Fact-checking via Relevant Evidence Detection](https://mlsanity.com/paper/2311.09939/) (2023)
- [Analysis and Comparison of Classification Metrics](https://mlsanity.com/paper/2209.05355/) (2022)
- [Multi-modal Misinformation Detection: Approaches, Challenges and Opportunities](https://mlsanity.com/paper/2203.13883/) (2022)
- [Human-in-the-loop Evaluation for Early Misinformation Detection: A Case Study of COVID-19 Treatments](https://mlsanity.com/paper/2212.09683/) (2022)
- [Similarity over Factuality: Are we making progress on multimodal out-of-context misinformation detection?](https://mlsanity.com/paper/2407.13488/) (2024)
- ["Image, Tell me your story!" Predicting the original meta-context of visual misinformation](https://mlsanity.com/paper/2408.09939/) (2024)
- [Learning Domain-Invariant Features for Out-of-Context News Detection](https://mlsanity.com/paper/2406.07430/) (2024)
- [Image-Text Out-Of-Context Detection Using Synthetic Multimodal Misinformation](https://mlsanity.com/paper/2403.08783/) (2024)
- [Multi-MLLM Knowledge Distillation for Out-of-Context News Detection](https://mlsanity.com/paper/2505.22517/) (2025)
- [Latent Reconstruction from Generated Data for Multimodal Misinformation Detection](https://mlsanity.com/paper/2504.06010/) (2025)
- [Is a Picture Worth a Thousand Words? Adaptive Multimodal Fact-Checking with Visual Evidence Necessity](https://mlsanity.com/paper/2604.04692/) (2026)
- [Evidence Triangulation for Multimodal Fact-Checking in the Wild](https://mlsanity.com/paper/2606.31367/) (2026)

## Abstract (arXiv)

Detecting out-of-context media, such as "mis-captioned" images on Twitter, is a relevant problem, especially in domains of high public significance. In this work we aim to develop defenses against such misinformation for the topics of Climate Change, COVID-19, and Military Vehicles. We first present a large-scale multimodal dataset with over 884k tweets relevant to these topics. Next, we propose a detection method, based on the state-of-the-art CLIP model, that leverages automatically generated hard image-text mismatches. While this approach works well on our automatically constructed out-of-context tweets, we aim to validate its usefulness on data representative of the real world. Thus, we test it on a set of human-generated fakes created by mimicking in-the-wild misinformation. We achieve an 11% detection improvement in a high precision regime over a strong baseline. Finally, we share insights about our best model design and analyze the challenges of this emerging threat.