mlsanityData to 7 Oct 2026

Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation

arXiv 2112.08594 ↗First posted 16 Dec 2021cs.CV cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Detecting out-of-context media, such as "mis-captioned" images on Twitter, is a relevant problem, especially in domains of high public significance. In this work we aim to develop defenses against such misinformation for the topics of Climate Change, COVID-19, and Military Vehicles. We first present a large-scale multimodal dataset with over 884k tweets relevant to these topics. Next, we propose a detection method, based on the state-of-the-art CLIP model, that leverages automatically generated hard image-text mismatches. While this approach works well on our automatically constructed out-of-context tweets, we aim to validate its usefulness on data representative of the real world. Thus, we test it on a set of human-generated fakes created by mimicking in-the-wild misinformation. We achieve an 11% detection improvement in a high precision regime over a strong baseline. Finally, we share insights about our best model design and analyze the challenges of this emerging threat.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 12 of its 29 references that are in the map. First posted on arXiv 16 Dec 2021.

Builds on

12 of 29 references.

References · 12

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  5. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  6. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
  7. 2021NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Mediacited by 106
  8. 2017Accelerated Hierarchical Density Clusteringcited by 100
  9. 2020Fighting an Infodemic: COVID-19 Fake News Datasetcited by 82
  10. 2020Multimodal Analytics for Real-world News using Measures of Cross-modal Entity Consistencycited by 33
  11. 2020A large-scale COVID-19 Twitter chatter dataset for open scientific research -- an international collaborationcited by 33
  12. 2017Multimedia Semantic Integrity Assessment Using Joint Embedding Of Images And Textcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 4; 2024: 5; 2025: 2; 2026: 2.

16 papers.

Sort:

  1. 2023Multimodal Automated Fact-Checking: A Surveycited by 51
  2. 2024DELL: Generating Reactions and Explanations for LLM-Based Misinformation Detectioncited by 41
  3. 2023VERITE: A Robust Benchmark for Multimodal Misinformation Detection Accounting for Unimodal Biascited by 37
  4. 2023Synthetic Misinformers: Generating and Combating Multimodal Misinformationcited by 23
  5. 2023RED-DOT: Multimodal Fact-checking via Relevant Evidence Detectioncited by 18
  6. 2022Analysis and Comparison of Classification Metricscited by 16
  7. 2022Multi-modal Misinformation Detection: Approaches, Challenges and Opportunitiescited by 14
  8. 2022Human-in-the-loop Evaluation for Early Misinformation Detection: A Case Study of COVID-19 Treatmentscited by 14
  9. 2024Similarity over Factuality: Are we making progress on multimodal out-of-context misinformation detection?cited by 11
  10. 2024Learning Domain-Invariant Features for Out-of-Context News Detectioncited by 3
  11. 2024"Image, Tell me your story!" Predicting the original meta-context of visual misinformationcited by 3
  12. 2024Image-Text Out-Of-Context Detection Using Synthetic Multimodal Misinformationcited by 2
4 more
  1. 2025Multi-MLLM Knowledge Distillation for Out-of-Context News Detectioncited by 2
  2. 2025Latent Reconstruction from Generated Data for Multimodal Misinformation Detectioncited by 1
  3. 2026Is a Picture Worth a Thousand Words? Adaptive Multimodal Fact-Checking with Visual Evidence Necessitycited by 0
  4. 2026Evidence Triangulation for Multimodal Fact-Checking in the Wildcited by 0