mlsanityData to 7 Oct 2026

Multimodal Fact-Checking with Vision Language Models: A Probing Classifier based Solution with Embedding Strategies

arXiv 2412.05155 ↗First posted 6 Dec 2024cs.CLcited by 10

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

This study evaluates the effectiveness of Vision Language Models (VLMs) in representing and utilizing multimodal content for fact-checking. To be more specific, we investigate whether incorporating multimodal content improves performance compared to text-only models and how well VLMs utilize text and image information to enhance misinformation detection. Furthermore we propose a probing classifier based solution using VLMs. Our approach extracts embeddings from the last hidden layer of selected VLMs and inputs them into a neural probing classifier for multi-class veracity classification. Through a series of experiments on two fact-checking datasets, we demonstrate that while multimodality can enhance performance, fusing separate embeddings from text and image encoders yielded superior results compared to using VLM embeddings. Furthermore, the proposed neural classifier significantly outperformed KNN and SVM baselines in leveraging extracted embeddings, highlighting its effectiveness for multimodal fact-checking.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 45 references that are in the map. First posted on arXiv 6 Dec 2024.

Builds on

20 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Multimodal Large Language Models to Support Real-World Fact-Checking6 Mar

Other references · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  5. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  6. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  7. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  8. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  9. 2024An Introduction to Vision-Language Modelingcited by 175
  10. 2023Fact-Checking Complex Claims with Program-Guided Reasoningcited by 146
  11. 2023AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Webcited by 144
  12. 2022End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Modelscited by 105
7 more
  1. 2023Multimodal Automated Fact-Checking: A Surveycited by 51
  2. 2018Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. Task 1: Check-Worthinesscited by 41
  3. 2023FactLLaMA: Optimizing Instruction-Following Language Models with External Knowledge for Automated Fact-Checkingcited by 37
  4. 2024JustiLM: Few-shot Justification Generation for Explainable Fact-Checking of Real-world Claimscited by 29
  5. 2023Team Triple-Check at Factify 2: Parameter-Efficient Large Foundation Models with Feature Representations for Multi-Modal Fact Verificationcited by 7
  6. 2024Exploring the Role of Audio in Multimodal Misinformation Detectioncited by 5
  7. 2024Cross-Lingual Learning vs. Low-Resource Fine-Tuning: A Case Study with Fact-Checking in Turkishcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 3.

10 papers.

  1. 2025The CLEF-2025 CheckThat! Lab: Subjectivity, Fact-Checking, Claim Normalization, and Retrievalcited by 14
  2. 2025Adversarial Attacks Against Automated Fact-Checking: A Surveycited by 5
  3. 2025mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generationcited by 4
  4. 2025MV-Debate: Multi-view Agent Debate with Dynamic Reflection Gating for Multimodal Harmful Content Detection in Social Mediacited by 4
  5. 2025HKD4VLM: A Progressive Hybrid Knowledge Distillation Framework for Robust Multimodal Hallucination and Factuality Detection in VLMscited by 3
  6. 2025Entailed Opinion Matters: Improving the Fact-Checking Performance of Language Models by Relying on their Entailment Abilitycited by 1
  7. 2025MultiCheck: Strengthening Web Trust with Unified Multimodal Fact Verificationcited by 1
  8. 2026Mitigating Object Hallucinations in LVLMs via Attention Imbalance Rectificationcited by 1
  9. 2026DANCE: Doubly Adaptive Neighborhood Conformal Estimationcited by 0
  10. 2026Is a Picture Worth a Thousand Words? Adaptive Multimodal Fact-Checking with Visual Evidence Necessitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MFC-Bench: Benchmarking Multimodal Fact-Checking with Large Vision-Language Models
  2. 2024Multimodal Misinformation Detection using Large Vision-Language Models
  3. 2024LEMMA: Towards LVLM-Enhanced Multimodal Misinformation Detection with External Knowledge Augmentation
  4. 2025E2LVLM:Evidence-Enhanced Large Vision-Language Model for Multimodal Out-of-Context Misinformation Detection
  5. 2025HintsOfTruth: A Multimodal Checkworthiness Detection Dataset with Real and Synthetic Claims
  6. 2025TRUST-VL: An Explainable News Assistant for General Multimodal Misinformation Detection
  7. 2026The Paradigm Shift: A Comprehensive Survey on Large Vision Language Models for Multimodal Fake News Detection
  8. 2025VLDBench Evaluating Multimodal Disinformation with Regulatory Alignment