# DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts

arXiv: https://arxiv.org/abs/2412.10510 · HTML: https://mlsanity.com/paper/2412.10510/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 92 references that are in the map.
- First posted on arXiv 13 Dec 2024.

## Builds on (most cited first, 30 of 64)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://mlsanity.com/paper/2204.00598/) (2022)
- ["Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection](https://mlsanity.com/paper/1705.00648/) (2017)
- [GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization](https://mlsanity.com/paper/2309.16020/) (2023)
- [Can LLM-Generated Misinformation Be Detected?](https://mlsanity.com/paper/2309.13788/) (2023)
- [Fact-Checking Complex Claims with Program-Guided Reasoning](https://mlsanity.com/paper/2305.12744/) (2023)
- [AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web](https://mlsanity.com/paper/2305.13117/) (2023)
- [Generating Fact Checking Explanations](https://mlsanity.com/paper/2004.05773/) (2020)
- [TruFor: Leveraging all-round clues for trustworthy image forgery detection and localization](https://mlsanity.com/paper/2212.10957/) (2022)
- [NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media](https://mlsanity.com/paper/2104.05893/) (2021)
- [End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models](https://mlsanity.com/paper/2205.12487/) (2022)
- [Detecting and Grounding Multi-Modal Media Manipulation](https://mlsanity.com/paper/2304.02556/) (2023)
- [Task Contamination: Language Models May Not Be Few-Shot Anymore](https://mlsanity.com/paper/2312.16337/) (2023)
- [Towards LLM-based Fact Verification on News Claims with a Hierarchical Step-by-Step Prompting Method](https://mlsanity.com/paper/2310.00305/) (2023)
- [Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers](https://mlsanity.com/paper/2311.09000/) (2023)
- [SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection](https://mlsanity.com/paper/2403.03170/) (2024)
- [Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resources](https://mlsanity.com/paper/2112.00061/) (2021)
- [Can ChatGPT Detect DeepFakes? A Study of Using Multimodal Large Language Models for Media Forensics](https://mlsanity.com/paper/2403.14077/) (2024)
- [MuMiN: A Large-Scale Multilingual Multimodal Fact-Checked Misinformation Social Network Dataset](https://mlsanity.com/paper/2202.11684/) (2022)
- [MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMs](https://mlsanity.com/paper/2406.08772/) (2024)
- [Multimodal Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2305.13507/) (2023)
- [Fact-Checking Meets Fauxtography: Verifying Claims About Images](https://mlsanity.com/paper/1908.11722/) (2019)
- [On the Origins of Memes by Means of Fringe Web Communities](https://mlsanity.com/paper/1805.12512/) (2018)
- [Benchmarks for Automated Commonsense Reasoning: A Survey](https://mlsanity.com/paper/2302.04752/) (2023)

## Built on by (most cited first, 30 of 43)

- [AVerImaTeC: A Dataset for Automatic Verification of Image-Text Claims with Evidence from the Web](https://mlsanity.com/paper/2505.17978/) (2025)
- [COVE: COntext and VEracity prediction for out-of-context images](https://mlsanity.com/paper/2502.01194/) (2025)
- [Fact-R1: Towards Explainable Video Misinformation Detection with Deep Reasoning](https://mlsanity.com/paper/2505.16836/) (2025)
- [M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset](https://mlsanity.com/paper/2510.23508/) (2025)
- [Debating Truth: Debate-driven Claim Verification with Multiple Large Language Model Agents](https://mlsanity.com/paper/2507.19090/) (2025)
- [Drifting Away from Truth: GenAI-Driven News Diversity Challenges LVLM-Based Misinformation Detection](https://mlsanity.com/paper/2508.12711/) (2025)
- [VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking](https://mlsanity.com/paper/2601.08611/) (2026)
- [Seeing Through Deception: Uncovering Misleading Creator Intent in Multimodal News with Vision-Language Models](https://mlsanity.com/paper/2505.15489/) (2025)
- [Adversarial Attacks Against Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2509.08463/) (2025)
- [T^2Agent A Tool-augmented Multimodal Misinformation Detection Agent with Monte Carlo Tree Search](https://mlsanity.com/paper/2505.19768/) (2025)
- [Multimedia Verification Through Multi-Agent Deep Research Multimodal Large Language Models](https://mlsanity.com/paper/2507.04410/) (2025)
- [Fact2Fiction: Targeted Poisoning Attack to Agentic Fact-checking System](https://mlsanity.com/paper/2508.06059/) (2025)
- [VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval](https://mlsanity.com/paper/2505.20291/) (2025)
- [Multi-Agent Retrieval-Augmented Framework for Evidence-Based Counterspeech Against Health Misinformation](https://mlsanity.com/paper/2507.07307/) (2025)
- [PASS-FC: Progressive and Adaptive Search Scheme for Fact Checking of Comprehensive Claims](https://mlsanity.com/paper/2504.09866/) (2025)
- [DECEIVE-AFC: Adversarial Claim Attacks against Search-Enabled LLM-based Fact-Checking Systems](https://mlsanity.com/paper/2602.02569/) (2026)
- [RW-Post: Auditable Evidence-Grounded Multimodal Fact-Checking in the Wild](https://mlsanity.com/paper/2605.10357/) (2026)
- [Beyond Retrieval: Improving Evidence Quality for LLM-based Multimodal Fact-Checking](https://mlsanity.com/paper/2505.03135/) (2025)
- [MultiCheck: Strengthening Web Trust with Unified Multimodal Fact Verification](https://mlsanity.com/paper/2508.05097/) (2025)
- [KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering](https://mlsanity.com/paper/2601.11632/) (2026)
- [CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrieval](https://mlsanity.com/paper/2601.17230/) (2026)
- [Fact or Fake? Assessing the Role of Deepfake Detectors in Multimodal Misinformation Detection](https://mlsanity.com/paper/2602.01854/) (2026)
- [Multimodal Claim Extraction for Fact-Checking](https://mlsanity.com/paper/2604.16311/) (2026)
- [When Misinformation Speaks and Converses: Rethinking Fact-Checking in Audio Platforms](https://mlsanity.com/paper/2604.16767/) (2026)
- [ClaimCheck: Real-Time Fact-Checking with Small Language Models](https://mlsanity.com/paper/2510.01226/) (2025)
- [Towards Real-Time Fake News Detection under Evidence Scarcity](https://mlsanity.com/paper/2510.11277/) (2025)
- [RW-Post: Auditable Evidence-Grounded Multimodal Fact-Checking in the Wild](https://mlsanity.com/paper/2512.22933/) (2025)
- [Multilingual, Multimodal Pipeline for Creating Authentic and Structured Fact-Checked Claim Dataset](https://mlsanity.com/paper/2601.07985/) (2026)
- [Frame-Guided Synthetic Claim Generation for Automatic Fact-Checking Using High-Volume Tabular Data](https://mlsanity.com/paper/2601.17232/) (2026)
- [Ostrakon-VL: Towards Domain-Expert MLLM for Food-Service and Retail Stores](https://mlsanity.com/paper/2601.21342/) (2026)

## Abstract (arXiv)

The proliferation of disinformation demands reliable and scalable fact-checking solutions. We present Dynamic Evidence-based FAct-checking with Multimodal Experts (DEFAME), a modular, zero-shot MLLM pipeline for open-domain, text-image claim verification. DEFAME operates in a six-stage process, dynamically selecting the tools and search depth to extract and evaluate textual and visual evidence. Unlike prior approaches that are text-only, lack explainability, or rely solely on parametric knowledge, DEFAME performs end-to-end verification, accounting for images in claims and evidence while generating structured, multimodal reports. Evaluation on the popular benchmarks VERITE, AVerITeC, and MOCHEG shows that DEFAME surpasses all previous methods, establishing itself as the new state-of-the-art fact-checking system for uni- and multimodal fact-checking. Moreover, we introduce a new multimodal benchmark, ClaimReview2024+, featuring claims after the knowledge cutoff of GPT-4o, avoiding data leakage. Here, DEFAME drastically outperforms the GPT-4o baselines, showing temporal generalizability and the potential for real-time fact-checking.