mlsanityData to 7 Oct 2026

End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models

arXiv 2205.12487 ↗First posted 25 May 2022cs.CLcited by 105

Not on the map. Why?

Abstract (arXiv)

We propose end-to-end multimodal fact-checking and explanation generation, where the input is a claim and a large collection of web sources, including articles, images, videos, and tweets, and the goal is to assess the truthfulness of the claim by retrieving relevant evidence and predicting a truthfulness label (e.g., support, refute or not enough information), and to generate a statement to summarize and explain the reasoning and ruling process. To support this research, we construct Mocheg, a large-scale dataset consisting of 15,601 claims where each claim is annotated with a truthfulness label and a ruling statement, and 33,880 textual paragraphs and 12,112 images in total as evidence. To establish baseline performances on Mocheg, we experiment with several state-of-the-art neural architectures on the three pipelined subtasks: multimodal evidence retrieval, claim verification, and explanation generation, and demonstrate that the performance of the state-of-the-art end-to-end multimodal fact-checking does not provide satisfactory outcomes. To the best of our knowledge, we are the first to build the benchmark dataset and solutions for end-to-end multimodal fact-checking and explanation generation. The dataset, source code and model checkpoints are available at https://github.com/VT-NLP/Mocheg.

Key facts. Cited by 105 papers in the map (data through 7 Oct 2026). Builds on 39 of its 85 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

39 of 85 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  9. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  10. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  11. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  12. 2021A Survey on Automated Fact-Checkingcited by 376
27 more
  1. 2017"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detectioncited by 361
  2. 2018FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Mediacited by 281
  3. 2020Explainable Automated Fact-Checking for Public Health Claimscited by 209
  4. 2019MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claimscited by 169
  5. 2021FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured informationcited by 166
  6. 2020Generating Fact Checking Explanationscited by 132
  7. 2018Combining Fact Extraction and Verification with Neural Semantic Matching Networkscited by 120
  8. 2019A Richly Annotated Corpus for Different Tasks in Automated Fact-Checkingcited by 74
  9. 2021Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resourcescited by 71
  10. 2022Generating Literal and Implied Subquestions to Fact-check Complex Claimscited by 66
  11. 2020FakeCovid -- A Multilingual Cross-domain Fact Check News Dataset for COVID-19cited by 60
  12. 2022MuMiN: A Large-Scale Multilingual Multimodal Fact-Checked Misinformation Social Network Datasetcited by 59
  13. 2019r/Fakeddit: A New Multimodal Benchmark Dataset for Fine-grained Fake News Detectioncited by 53
  14. 2019Fact-Checking Meets Fauxtography: Verifying Claims About Imagescited by 52
  15. 2020Generating Fact Checking Briefscited by 50
  16. 2020The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizescited by 48
  17. 2020Detecting Cross-Modal Inconsistency to Defend Against Neural Fake Newscited by 41
  18. 2022PLANET: Dynamic Content Planning in Autoregressive Transformers for Long-form Text Generationcited by 30
  19. 2021Explain and Predict, and then Predict Againcited by 29
  20. 2021Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfercited by 28
  21. 2019NELA-GT-2018: A Large Multi-Labelled News Dataset for The Study of Misinformation in News Articlescited by 26
  22. 2016A Compare-Aggregate Model for Matching Text Sequencescited by 23
  23. 2021Multimodal Emergent Fake News Detection via Meta Neural Process Networkscited by 21
  24. 2021Explainable Fact-checking through Question Answeringcited by 16
  25. 2020A Dataset of Fact-Checked Images Shared on WhatsApp During the Brazilian and Indian Electionscited by 9
  26. 2020A Multi-Modal Method for Satire Detection using Textual and Visual Cuescited by 9
  27. 2021Extractive and Abstractive Explanations for Fact-Checking and Evaluation of Newscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 13; 2024: 35; 2025: 31; 2026: 25.

105 papers.

Sort:

By area:

  1. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  2. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  3. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  4. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  5. 2024MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMscited by 60
  6. 2025A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challengescited by 58
  7. 2023Multimodal Automated Fact-Checking: A Surveycited by 51
  8. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  9. 2024DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Expertscited by 42
  10. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  11. 2023Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checkingcited by 34
  12. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31
38 more
  1. 2024JustiLM: Few-shot Justification Generation for Explainable Fact-Checking of Real-world Claimscited by 29
  2. 2025Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievalscited by 27
  3. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  4. 2024Retrieval Augmented Fact Verification by Synthesizing Contrastive Argumentscited by 25
  5. 2023IRFL: Image Recognition of Figurative Languagecited by 23
  6. 2023Factify 2: A Multimodal Fake News and Satire News Datasetcited by 23
  7. 2023Support or Refute: Analyzing the Stance of Evidence to Detect Out-of-Context Mis- and Disinformationcited by 21
  8. 2024Multimodal Misinformation Detection using Large Vision-Language Modelscited by 21
  9. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 21
  10. 2024Fairness and Bias in Multimodal AI: A Surveycited by 19
  11. 2023RED-DOT: Multimodal Fact-checking via Relevant Evidence Detectioncited by 18
  12. 2024Can LLMs Improve Multimodal Fact-Checking by Asking Relevant Questions?cited by 17
  13. 2024CAMEL-Bench: A Comprehensive Arabic LMM Benchmarkcited by 17
  14. 2025AVerImaTeC: A Dataset for Automatic Verification of Image-Text Claims with Evidence from the Webcited by 17
  15. 2023FACTIFY3M: A Benchmark for Multimodal Fact Verification with Explainability through 5W Question-Answeringcited by 15
  16. 2024MFC-Bench: Benchmarking Multimodal Fact-Checking with Large Vision-Language Modelscited by 15
  17. 2024Automated Justification Production for Claim Veracity in Fact Checking: A Survey on Architectures and Approachescited by 15
  18. 2024Multimodal Large Language Models to Support Real-World Fact-Checkingcited by 13
  19. 2024Multimodal Misinformation Detection by Learning from Synthetic Data with Multimodal LLMscited by 13
  20. 2025Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Modelscited by 13
  21. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarkscited by 12
  22. 2024Piecing It All Together: Verifying Multi-Hop Multimodal Claimscited by 12
  23. 2024Similarity over Factuality: Are we making progress on multimodal out-of-context misinformation detection?cited by 11
  24. 2024CommunityKG-RAG: Leveraging Community Structures in Knowledge Graphs for Advanced Retrieval-Augmented Generation in Fact-Checkingcited by 11
  25. 2024Each Fake News is Fake in its Own Way: An Attribution Multi-Granularity Benchmark for Multimodal Fake News Detectioncited by 11
  26. 2024MetaSumPerceiver: Multimodal Multi-Document Evidence Summarization for Fact-Checkingcited by 10
  27. 2024Multimodal Fact-Checking with Vision Language Models: A Probing Classifier based Solution with Embedding Strategiescited by 10
  28. 2025TRUST-VL: An Explainable News Assistant for General Multimodal Misinformation Detectioncited by 10
  29. 2025AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understandingcited by 9
  30. 2024Credible, Unreliable or Leaked?: Evidence Verification for Enhanced Automated Fact-checkingcited by 8
  31. 2025The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Paperscited by 6
  32. 2025XFacta: Contemporary, Real-World Dataset and Evaluation for Multimodal Misinformation Detection with Multimodal LLMscited by 6
  33. 2026The Paradigm Shift: A Comprehensive Survey on Large Vision Language Models for Multimodal Fake News Detectioncited by 6
  34. 2024LiveFC: A System for Live Fact-Checking of Audio Streamscited by 5
  35. 2024Face the Facts! Evaluating RAG-based Pipelines for Professional Fact-Checkingcited by 5
  36. 2025Towards Automated Fact-Checking of Real-World Claims: Exploring Task Formulation and Assessment with LLMscited by 5
  37. 2025Large Language Models Meet Stance Detection: A Survey of Tasks, Methods, Applications, Challenges and Future Directionscited by 5
  38. 2025Seeing Through Deception: Uncovering Misleading Creator Intent in Multimodal News with Vision-Language Modelscited by 5

50 most cited of 105 All 105 (JSON)