# MetaSumPerceiver: Multimodal Multi-Document Evidence Summarization for Fact-Checking

arXiv: https://arxiv.org/abs/2407.13089 · HTML: https://mlsanity.com/paper/2407.13089/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 18 Jul 2024.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019)
- [Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations](https://mlsanity.com/paper/2301.04246/) (2023)
- [Co-Writing with Opinionated Language Models Affects Users' Views](https://mlsanity.com/paper/2302.00560/) (2023)
- [End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models](https://mlsanity.com/paper/2205.12487/) (2022)
- [Human heuristics for AI-generated language are flawed](https://mlsanity.com/paper/2206.07271/) (2022)
- [AI model GPT-3 (dis)informs us better than humans](https://mlsanity.com/paper/2301.11924/) (2023)
- [MuMiN: A Large-Scale Multilingual Multimodal Fact-Checked Misinformation Social Network Dataset](https://mlsanity.com/paper/2202.11684/) (2022)
- [Scientific Fact-Checking: A Survey of Resources and Approaches](https://mlsanity.com/paper/2305.16859/) (2023)
- [Fact-Checking Meets Fauxtography: Verifying Claims About Images](https://mlsanity.com/paper/1908.11722/) (2019)
- [Data-to-text Generation with Macro Planning](https://mlsanity.com/paper/2102.02723/) (2021)
- [Robust (Controlled) Table-to-Text Generation with Structure-Aware Equivariance Learning](https://mlsanity.com/paper/2205.03972/) (2022)
- [A Benchmark Dataset of Check-worthy Factual Claims](https://mlsanity.com/paper/2004.14425/) (2020)
- [Overview of the CLEF--2021 CheckThat! Lab on Detecting Check-Worthy Claims, Previously Fact-Checked Claims, and Fake News](https://mlsanity.com/paper/2109.12987/) (2021)
- [Long Text and Multi-Table Summarization: Dataset and Method](https://mlsanity.com/paper/2302.03815/) (2023)

## Built on by (most cited first, 10 of 10)

- [DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts](https://mlsanity.com/paper/2412.10510/) (2024)
- [On the Risk of Evidence Pollution for Malicious Social Text Detection in the Era of LLMs](https://mlsanity.com/paper/2410.12600/) (2024)
- [Uncovering the Bigger Picture: Comprehensive Event Understanding Via Diverse News Retrieval](https://mlsanity.com/paper/2508.19758/) (2025)
- [How Private are Language Models in Abstractive Summarization?](https://mlsanity.com/paper/2412.12040/) (2024)
- [RW-Post: Auditable Evidence-Grounded Multimodal Fact-Checking in the Wild](https://mlsanity.com/paper/2605.10357/) (2026)
- [Exploring Modality Disruption in Multimodal Fake News Detection](https://mlsanity.com/paper/2504.09154/) (2025)
- [ABCD-LINK: Annotation Bootstrapping for Cross-Document Fine-Grained Links](https://mlsanity.com/paper/2509.01387/) (2025)
- [RW-Post: Auditable Evidence-Grounded Multimodal Fact-Checking in the Wild](https://mlsanity.com/paper/2512.22933/) (2025)
- [Global Consistency Repair for Natural-Language Claim Sets](https://mlsanity.com/paper/2601.13600/) (2026)
- [Is a Picture Worth a Thousand Words? Adaptive Multimodal Fact-Checking with Visual Evidence Necessity](https://mlsanity.com/paper/2604.04692/) (2026)

## Abstract (arXiv)

Fact-checking real-world claims often requires reviewing multiple multimodal documents to assess a claim's truthfulness, which is a highly laborious and time-consuming task. In this paper, we present a summarization model designed to generate claim-specific summaries useful for fact-checking from multimodal, multi-document datasets. The model takes inputs in the form of documents, images, and a claim, with the objective of assisting in fact-checking tasks. We introduce a dynamic perceiver-based model that can handle inputs from multiple modalities of arbitrary lengths. To train our model, we leverage a novel reinforcement learning-based entailment objective to generate summaries that provide evidence distinguishing between different truthfulness labels. To assess the efficacy of our approach, we conduct experiments on both an existing benchmark and a new dataset of multi-document claims that we contribute. Our approach outperforms the SOTA approach by 4.6% in the claim verification task on the MOCHEG dataset and demonstrates strong performance on our new Multi-News-Fact-Checking dataset.