# Can LLMs Improve Multimodal Fact-Checking by Asking Relevant Questions?

arXiv: https://arxiv.org/abs/2410.04616 · HTML: https://mlsanity.com/paper/2410.04616/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 69 references that are in the map.
- First posted on arXiv 6 Oct 2024.

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)
- [VILA: On Pre-training for Visual Language Models](https://mlsanity.com/paper/2312.07533/) (2023)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Media](https://mlsanity.com/paper/1809.01286/) (2018)
- [Can LLM-Generated Misinformation Be Detected?](https://mlsanity.com/paper/2309.13788/) (2023)
- [Long-form factuality in large language models](https://mlsanity.com/paper/2403.18802/) (2024)
- [Automated Fact-Checking for Assisting Human Fact-Checkers](https://mlsanity.com/paper/2103.07769/) (2021)
- [Visual News: Benchmark and Challenges in News Image Captioning](https://mlsanity.com/paper/2010.03743/) (2020)
- [Combining Fact Extraction and Verification with Neural Semantic Matching Networks](https://mlsanity.com/paper/1811.07039/) (2018)
- [End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models](https://mlsanity.com/paper/2205.12487/) (2022)
- [Detecting and Grounding Multi-Modal Media Manipulation](https://mlsanity.com/paper/2304.02556/) (2023)
- [Towards LLM-based Fact Verification on News Claims with a Hierarchical Step-by-Step Prompting Method](https://mlsanity.com/paper/2310.00305/) (2023)
- [Generating Literal and Implied Subquestions to Fact-check Complex Claims](https://mlsanity.com/paper/2205.06938/) (2022)
- [MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMs](https://mlsanity.com/paper/2406.08772/) (2024)
- [r/Fakeddit: A New Multimodal Benchmark Dataset for Fine-grained Fake News Detection](https://mlsanity.com/paper/1911.03854/) (2019)
- [Multimodal Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2305.13507/) (2023)
- [Generating Fact Checking Briefs](https://mlsanity.com/paper/2011.05448/) (2020)
- [Disinformation Detection: An Evolving Challenge in the Age of LLMs](https://mlsanity.com/paper/2309.15847/) (2023)
- [MM-COVID: A Multilingual and Multimodal Data Repository for Combating COVID-19 Disinformation](https://mlsanity.com/paper/2011.04088/) (2020)
- [Fully Automated Fact Checking Using External Sources](https://mlsanity.com/paper/1710.00341/) (2017)
- [QACHECK: A Demonstration System for Question-Guided Multi-Hop Fact-Checking](https://mlsanity.com/paper/2310.07609/) (2023)
- [Interpretable Multimodal Misinformation Detection with Logic Reasoning](https://mlsanity.com/paper/2305.05964/) (2023)

## Built on by (most cited first, 17 of 17)

- [Preference Leakage: A Contamination Problem in LLM-as-a-judge](https://mlsanity.com/paper/2502.01534/) (2025)
- [SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agents](https://mlsanity.com/paper/2411.03284/) (2024)
- [Drifting Away from Truth: GenAI-Driven News Diversity Challenges LVLM-Based Misinformation Detection](https://mlsanity.com/paper/2508.12711/) (2025)
- [T^2Agent A Tool-augmented Multimodal Misinformation Detection Agent with Monte Carlo Tree Search](https://mlsanity.com/paper/2505.19768/) (2025)
- [What's Left Unsaid? Detecting and Correcting Misleading Omissions in Multimodal News Previews](https://mlsanity.com/paper/2601.05563/) (2026)
- [Assessing the Impact of Conspiracy Theories Using Large Language Models](https://mlsanity.com/paper/2412.07019/) (2024)
- [MNIST-Gen: A Modular MNIST-Style Dataset Generation Using Hierarchical Semantics, Reinforcement Learning, and Category Theory](https://mlsanity.com/paper/2507.11821/) (2025)
- [Explainable Human-in-the-Loop Segmentation via Critic Feedback Signals](https://mlsanity.com/paper/2510.09945/) (2025)
- [A parameter-efficient three-branch architecture for multimodal misinformation detection with limited annotations](https://mlsanity.com/paper/2512.20257/) (2025)
- [Fact or Fake? Assessing the Role of Deepfake Detectors in Multimodal Misinformation Detection](https://mlsanity.com/paper/2602.01854/) (2026)
- [Probing to Refine: Reinforcement Distillation of LLMs via Explanatory Inversion](https://mlsanity.com/paper/2603.19266/) (2026)
- [Tri-Accel: Curvature-Aware Precision-Adaptive and Memory-Elastic Optimization for Efficient GPU Usage](https://mlsanity.com/paper/2508.16905/) (2025)
- [Insight-A: Attribution-aware for Multimodal Misinformation Detection](https://mlsanity.com/paper/2511.21705/) (2025)
- [Agentic Mixed-Source Multi-Modal Misinformation Detection with Adaptive Test-Time Scaling](https://mlsanity.com/paper/2603.02519/) (2026)
- [ReMMD: Realistic Multilingual Multi-Image Agentic Verification for Multimodal Misinformation Detection](https://mlsanity.com/paper/2606.24112/) (2026)
- [Verification-Notebook Learning for Source-Aware Multimodal Misinformation Detection](https://mlsanity.com/paper/2607.23581/) (2026)
- [Vis-Poison: Poisoning Visual Knowledge in Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2608.20756/) (2026)

## Abstract (arXiv)

Traditional fact-checking relies on humans to formulate relevant and targeted fact-checking questions (FCQs), search for evidence, and verify the factuality of claims. While Large Language Models (LLMs) have been commonly used to automate evidence retrieval and factuality verification at scale, their effectiveness for fact-checking is hindered by the absence of FCQ formulation. To bridge this gap, we seek to answer two research questions: (1) Can LLMs generate relevant FCQs? (2) Can LLM-generated FCQs improve multimodal fact-checking? We therefore introduce a framework LRQ-FACT for using LLMs to generate relevant FCQs to facilitate evidence retrieval and enhance fact-checking by probing information across multiple modalities. Through extensive experiments, we verify if LRQ-FACT can generate relevant FCQs of different types and if LRQ-FACT can consistently outperform baseline methods in multimodal fact-checking. Further analysis illustrates how each component in LRQ-FACT works toward improving the fact-checking performance.