# VERITE: A Robust Benchmark for Multimodal Misinformation Detection Accounting for Unimodal Bias

arXiv: https://arxiv.org/abs/2304.14133 · HTML: https://mlsanity.com/paper/2304.14133/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 59 references that are in the map.
- First posted on arXiv 27 Apr 2023.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering](https://mlsanity.com/paper/1712.00377/) (2017)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [RUBi: Reducing Unimodal Biases in Visual Question Answering](https://mlsanity.com/paper/1906.10169/) (2019)
- [Frozen CLIP Models are Efficient Video Learners](https://mlsanity.com/paper/2208.03550/) (2022)
- [Visual News: Benchmark and Challenges in News Image Captioning](https://mlsanity.com/paper/2010.03743/) (2020)
- [NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media](https://mlsanity.com/paper/2104.05893/) (2021)
- [CLIP-Event: Connecting Text and Images with Event Structures](https://mlsanity.com/paper/2201.05078/) (2022)
- [Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resources](https://mlsanity.com/paper/2112.00061/) (2021)
- [MuMiN: A Large-Scale Multilingual Multimodal Fact-Checked Misinformation Social Network Dataset](https://mlsanity.com/paper/2202.11684/) (2022)
- [r/Fakeddit: A New Multimodal Benchmark Dataset for Fine-grained Fake News Detection](https://mlsanity.com/paper/1911.03854/) (2019)
- [Fact-Checking Meets Fauxtography: Verifying Claims About Images](https://mlsanity.com/paper/1908.11722/) (2019)
- [A Survey on Multimodal Disinformation Detection](https://mlsanity.com/paper/2103.12541/) (2021)
- [Multimodal Fake News Detection via CLIP-Guided Learning](https://mlsanity.com/paper/2205.14304/) (2022)
- [Multimodal Analytics for Real-world News using Measures of Cross-modal Entity Consistency](https://mlsanity.com/paper/2003.10421/) (2020)
- [Deep Multimodal Image-Repurposing Detection](https://mlsanity.com/paper/1808.06686/) (2018)
- [Multimedia Semantic Integrity Assessment Using Joint Embedding Of Images And Text](https://mlsanity.com/paper/1707.01606/) (2017)
- [COSMOS: Catching Out-of-Context Misinformation with Self-Supervised Learning](https://mlsanity.com/paper/2101.06278/) (2021)
- [Synthetic Misinformers: Generating and Combating Multimodal Misinformation](https://mlsanity.com/paper/2303.01217/) (2023)
- [The PS-Battles Dataset - an Image Collection for Image Manipulation Detection](https://mlsanity.com/paper/1804.04866/) (2018)
- [Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation](https://mlsanity.com/paper/2112.08594/) (2021)
- [Identifying Nuances in Fake News vs. Satire: Using Semantic and Linguistic Cues](https://mlsanity.com/paper/1910.01160/) (2019)
- [MMSys'21 Grand Challenge on Detecting Cheapfakes](https://mlsanity.com/paper/2107.05297/) (2021)
- [MM-Claims: A Dataset for Multimodal Claim Detection in Social Media](https://mlsanity.com/paper/2205.01989/) (2022)

## Built on by (most cited first, 30 of 39)

- [Detecting Multimedia Generated by Large AI Models: A Survey](https://mlsanity.com/paper/2402.00045/) (2024)
- [SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection](https://mlsanity.com/paper/2403.03170/) (2024)
- [DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts](https://mlsanity.com/paper/2412.10510/) (2024)
- [RED-DOT: Multimodal Fact-checking via Relevant Evidence Detection](https://mlsanity.com/paper/2311.09939/) (2023)
- [AVerImaTeC: A Dataset for Automatic Verification of Image-Text Claims with Evidence from the Web](https://mlsanity.com/paper/2505.17978/) (2025)
- [Similarity over Factuality: Are we making progress on multimodal out-of-context misinformation detection?](https://mlsanity.com/paper/2407.13488/) (2024)
- [A Guide to Misinformation Detection Data and Evaluation](https://mlsanity.com/paper/2411.05060/) (2024)
- [TRUST-VL: An Explainable News Assistant for General Multimodal Misinformation Detection](https://mlsanity.com/paper/2509.04448/) (2025)
- [Credible, Unreliable or Leaked?: Evidence Verification for Enhanced Automated Fact-checking](https://mlsanity.com/paper/2404.18971/) (2024)
- [E2LVLM:Evidence-Enhanced Large Vision-Language Model for Multimodal Out-of-Context Misinformation Detection](https://mlsanity.com/paper/2502.10455/) (2025)
- [XFacta: Contemporary, Real-World Dataset and Evaluation for Multimodal Misinformation Detection with Multimodal LLMs](https://mlsanity.com/paper/2508.09999/) (2025)
- [M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset](https://mlsanity.com/paper/2510.23508/) (2025)
- [CMIE: Combining MLLM Insights with External Evidence for Explainable Out-of-Context Misinformation Detection](https://mlsanity.com/paper/2505.23449/) (2025)
- [The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Papers](https://mlsanity.com/paper/2506.20844/) (2025)
- [Drifting Away from Truth: GenAI-Driven News Diversity Challenges LVLM-Based Misinformation Detection](https://mlsanity.com/paper/2508.12711/) (2025)
- [VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking](https://mlsanity.com/paper/2601.08611/) (2026)
- [The Paradigm Shift: A Comprehensive Survey on Large Vision Language Models for Multimodal Fake News Detection](https://mlsanity.com/paper/2601.15316/) (2026)
- [FactGenius: Combining Zero-Shot Prompting and Fuzzy Relation Mining to Improve Fact Verification with Knowledge Graphs](https://mlsanity.com/paper/2406.01311/) (2024)
- [MAD-Sherlock: Multi-Agent Debate for Visual Misinformation Detection](https://mlsanity.com/paper/2410.20140/) (2024)
- [HintsOfTruth: A Multimodal Checkworthiness Detection Dataset with Real and Synthetic Claims](https://mlsanity.com/paper/2502.11753/) (2025)
- [A Self-Learning Multimodal Approach for Fake News Detection](https://mlsanity.com/paper/2412.05843/) (2024)
- ["Humor, Art, or Misinformation?": A Multimodal Dataset for Intent-Aware Synthetic Image Detection](https://mlsanity.com/paper/2508.20670/) (2025)
- [The Automatic Verification of Image-Text Claims (AVerImaTeC) Shared Task](https://mlsanity.com/paper/2602.11221/) (2026)
- [The Synthetic Media Shift: Tracking the Rise, Virality, and Detectability of AI-Generated Multimodal Misinformation](https://mlsanity.com/paper/2604.15372/) (2026)
- [Can Out-of-Domain data help to Learn Domain-Specific Prompts for Multimodal Misinformation Detection?](https://mlsanity.com/paper/2311.16496/) (2023)
- [Mitigating GenAI-Powered Evidence Pollution for Out-Of-Context Misinformation Detection](https://mlsanity.com/paper/2501.14728/) (2025)
- [Latent Reconstruction from Generated Data for Multimodal Misinformation Detection](https://mlsanity.com/paper/2504.06010/) (2025)
- [Unveiling Modality Bias: Automated Sample-Specific Analysis for Multimodal Misinformation Benchmarks](https://mlsanity.com/paper/2511.05883/) (2025)
- [A parameter-efficient three-branch architecture for multimodal misinformation detection with limited annotations](https://mlsanity.com/paper/2512.20257/) (2025)
- [MultiCaption: Detecting disinformation using multilingual visual claims](https://mlsanity.com/paper/2601.11220/) (2026)

## Abstract (arXiv)

Multimedia content has become ubiquitous on social media platforms, leading to the rise of multimodal misinformation (MM) and the urgent need for effective strategies to detect and prevent its spread. In recent years, the challenge of multimodal misinformation detection (MMD) has garnered significant attention by researchers and has mainly involved the creation of annotated, weakly annotated, or synthetically generated training datasets, along with the development of various deep learning MMD models. However, the problem of unimodal bias has been overlooked, where specific patterns and biases in MMD benchmarks can result in biased or unimodal models outperforming their multimodal counterparts on an inherently multimodal task; making it difficult to assess progress. In this study, we systematically investigate and identify the presence of unimodal bias in widely-used MMD benchmarks, namely VMU-Twitter and COSMOS. To address this issue, we introduce the "VERification of Image-TExt pairs" (VERITE) benchmark for MMD which incorporates real-world data, excludes "asymmetric multimodal misinformation" and utilizes "modality balancing". We conduct an extensive comparative study with a Transformer-based architecture that shows the ability of VERITE to effectively address unimodal bias, rendering it a robust evaluation framework for MMD. Furthermore, we introduce a new method -- termed Crossmodal HArd Synthetic MisAlignment (CHASMA) -- for generating realistic synthetic training data that preserve crossmodal relations between legitimate images and false human-written captions. By leveraging CHASMA in the training process, we observe consistent and notable improvements in predictive performance on VERITE; with a 9.2% increase in accuracy. We release our code at: https://github.com/stevejpapad/image-text-verification