mlsanityData to 7 Oct 2026

NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media

arXiv 2104.05893 ↗First posted 13 Apr 2021cs.CV cs.CLcited by 106

Not on the map. Why?

Abstract (arXiv)

Online misinformation is a prevalent societal issue, with adversaries relying on tools ranging from cheap fakes to sophisticated deep fakes. We are motivated by the threat scenario where an image is used out of context to support a certain narrative. While some prior datasets for detecting image-text inconsistency generate samples via text manipulation, we propose a dataset where both image and text are unmanipulated but mismatched. We introduce several strategies for automatically retrieving convincing images for a given caption, capturing cases with inconsistent entities or semantic context. Our large-scale automatically generated NewsCLIPpings Dataset: (1) demonstrates that machine-driven image repurposing is now a realistic threat, and (2) provides samples that represent challenging instances of mismatch between text and image in news that are able to mislead humans. We benchmark several state-of-the-art multimodal models on our dataset and analyze their performance across different pretraining domains and visual backbones.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Builds on 18 of its 27 references that are in the map. First posted on arXiv 13 Apr 2021.

Builds on

18 of 27 references.

References · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  6. 2019Defending Against Neural Fake Newscited by 553
  7. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  8. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  9. 2018FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Mediacited by 281
  10. 2019Measuring Bias in Contextualized Word Representationscited by 275
  11. 2020REL: An Entity Linker Standing on the Shoulders of Giantscited by 61
  12. 2020SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Modelscited by 55
6 more
  1. 2020Detecting Cross-Modal Inconsistency to Defend Against Neural Fake Newscited by 41
  2. 2020Multimodal Analytics for Real-world News using Measures of Cross-modal Entity Consistencycited by 33
  3. 2018Deep Multimodal Image-Repurposing Detectioncited by 31
  4. 2017Multimedia Semantic Integrity Assessment Using Joint Embedding Of Images And Textcited by 28
  5. 2021COSMOS: Catching Out-of-Context Misinformation with Self-Supervised Learningcited by 26
  6. 2020Preserving Semantic Neighborhoods for Robust Cross-modal Retrievalcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 3; 2023: 15; 2024: 21; 2025: 36; 2026: 27.

106 papers.

Sort:

By area:

  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  2. 2023Detecting and Grounding Multi-Modal Media Manipulationcited by 98
  3. 2024Detecting Multimedia Generated by Large AI Models: A Surveycited by 73
  4. 2024SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detectioncited by 73
  5. 2021Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resourcescited by 71
  6. 2024MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMscited by 60
  7. 2023Detecting and Grounding Multi-Modal Media Manipulation and Beyondcited by 55
  8. 2023Multimodal Automated Fact-Checking: A Surveycited by 51
  9. 2024DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Expertscited by 42
  10. 2024FKA-Owl: Advancing Multimodal Fake News Detection through Knowledge-Augmented LVLMscited by 38
  11. 2023VERITE: A Robust Benchmark for Multimodal Misinformation Detection Accounting for Unimodal Biascited by 37
  12. 2023Synthetic Misinformers: Generating and Combating Multimodal Misinformationcited by 23
38 more
  1. 2023Support or Refute: Analyzing the Stance of Evidence to Detect Out-of-Context Mis- and Disinformationcited by 21
  2. 2023RED-DOT: Multimodal Fact-checking via Relevant Evidence Detectioncited by 18
  3. 2025AVerImaTeC: A Dataset for Automatic Verification of Image-Text Claims with Evidence from the Webcited by 17
  4. 2021Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformationcited by 16
  5. 2023COVID-VTS: Fact Extraction and Verification on Short Video Platformscited by 15
  6. 2023FACTIFY3M: A Benchmark for Multimodal Fact Verification with Explainability through 5W Question-Answeringcited by 15
  7. 2024MFC-Bench: Benchmarking Multimodal Fact-Checking with Large Vision-Language Modelscited by 15
  8. 2023Unified Frequency-Assisted Transformer Framework for Detecting and Grounding Multi-Modal Manipulationcited by 14
  9. 2025COVE: COntext and VEracity prediction for out-of-context imagescited by 14
  10. 2025Fact-R1: Towards Explainable Video Misinformation Detection with Deep Reasoningcited by 14
  11. 2023ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representationcited by 13
  12. 2024Multimodal Large Language Models to Support Real-World Fact-Checkingcited by 13
  13. 2024Multimodal Misinformation Detection by Learning from Synthetic Data with Multimodal LLMscited by 13
  14. 2025Forensics-Bench: A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Modelscited by 13
  15. 2025Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Modelscited by 13
  16. 2024Piecing It All Together: Verifying Multi-Hop Multimodal Claimscited by 12
  17. 2022Fact-Saboteurs: A Taxonomy of Evidence Manipulation Attacks against Fact-Verification Systemscited by 11
  18. 2024Similarity over Factuality: Are we making progress on multimodal out-of-context misinformation detection?cited by 11
  19. 2024Each Fake News is Fake in its Own Way: An Attribution Multi-Granularity Benchmark for Multimodal Fake News Detectioncited by 11
  20. 2021Multi-Modal Semantic Inconsistency Detection in Social Media News Postscited by 10
  21. 2023Robust Domain Misinformation Detection via Multi-modal Feature Alignmentcited by 10
  22. 2025TRUST-VL: An Explainable News Assistant for General Multimodal Misinformation Detectioncited by 10
  23. 2023Interpretable Detection of Out-of-Context Misinformation with Neural-Symbolic-Enhanced Large Multimodal Modelcited by 8
  24. 2024Credible, Unreliable or Leaked?: Evidence Verification for Enhanced Automated Fact-checkingcited by 8
  25. 2025VLDBench Evaluating Multimodal Disinformation with Regulatory Alignmentcited by 8
  26. 2025The Coherence Trap: When MLLM-Crafted Narratives Exploit Manipulated Visual Contextscited by 8
  27. 2025M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Datasetcited by 7
  28. 2025E2LVLM:Evidence-Enhanced Large Vision-Language Model for Multimodal Out-of-Context Misinformation Detectioncited by 6
  29. 2025CMIE: Combining MLLM Insights with External Evidence for Explainable Out-of-Context Misinformation Detectioncited by 6
  30. 2025XFacta: Contemporary, Real-World Dataset and Evaluation for Multimodal Misinformation Detection with Multimodal LLMscited by 6
  31. 2025Drifting Away from Truth: GenAI-Driven News Diversity Challenges LVLM-Based Misinformation Detectioncited by 6
  32. 2026The Paradigm Shift: A Comprehensive Survey on Large Vision Language Models for Multimodal Fake News Detectioncited by 6
  33. 2025Verifying Cross-modal Entity Consistency in News using Vision-language Modelscited by 5
  34. 2025Seeing Through Deception: Uncovering Misleading Creator Intent in Multimodal News with Vision-Language Modelscited by 5
  35. 2025Unleashing the Potential of Consistency Learning for Detecting and Grounding Multi-Modal Media Manipulationcited by 5
  36. 2025RAMA: Retrieval-Augmented Multi-Agent Framework for Misinformation Detection in Multimodal Fact-Checkingcited by 5
  37. 2026VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checkingcited by 5
  38. 2022Misinformation Detection in Social Media Video Postscited by 4

50 most cited of 106 All 106 (JSON)