# Fine-grained and Explainable Factuality Evaluation for Multimodal Summarization

arXiv: https://arxiv.org/abs/2402.11414 · HTML: https://mlsanity.com/paper/2402.11414/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 38 references that are in the map.
- First posted on arXiv 18 Feb 2024.

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization](https://mlsanity.com/paper/2005.03754/) (2020)
- [QuestEval: Summarization Asks for Fact-based Evaluation](https://mlsanity.com/paper/2103.12693/) (2021)
- [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://mlsanity.com/paper/2112.08542/) (2021)
- [CREPE: Can Vision-Language Foundation Models Reason Compositionally?](https://mlsanity.com/paper/2212.07796/) (2022)
- [Annotating and Modeling Fine-grained Factuality in Summarization](https://mlsanity.com/paper/2104.04302/) (2021)
- [VMSMO: Learning to Generate Multimodal Summary for Video-based News Articles](https://mlsanity.com/paper/2010.05406/) (2020)
- [CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning](https://mlsanity.com/paper/2112.08713/) (2021)
- [Multimodal Abstractive Summarization for How2 Videos](https://mlsanity.com/paper/1906.07901/) (2019)
- [Factual Consistency Evaluation for Text Summarization via Counterfactual Estimation](https://mlsanity.com/paper/2108.13134/) (2021)
- [Hierarchical Cross-Modality Semantic Correlation Learning Model for Multimodal Summarization](https://mlsanity.com/paper/2112.12072/) (2021)
- [Exploiting Pseudo Image Captions for Multimodal Summarization](https://mlsanity.com/paper/2305.05496/) (2023)
- [Evaluating and Improving Factuality in Multimodal Abstractive Summarization](https://mlsanity.com/paper/2211.02580/) (2022)

## Built on by (most cited first, 10 of 10)

- [FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedback](https://mlsanity.com/paper/2404.05046/) (2024)
- [Can LLMs Refuse Questions They Do Not Know? Measuring Knowledge-Aware Refusal in Factual Tasks](https://mlsanity.com/paper/2510.01782/) (2025)
- [Can Large Vision-Language Models Understand Multimodal Sarcasm?](https://mlsanity.com/paper/2508.03654/) (2025)
- [A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2505.01958/) (2025)
- [Long-Form Information Alignment Evaluation Beyond Atomic Facts](https://mlsanity.com/paper/2505.15792/) (2025)
- [FIHA: Autonomous Hallucination Evaluation in Vision-Language Models with Davidson Scene Graphs](https://mlsanity.com/paper/2409.13612/) (2024)
- [A Unified Hallucination Mitigation Framework for Large Vision-Language Models](https://mlsanity.com/paper/2409.16494/) (2024)
- [Defeasible Visual Entailment: Benchmark, Evaluator, and Reward-Driven Optimization](https://mlsanity.com/paper/2412.16232/) (2024)
- [Integrating Video and Text: A Balanced Approach to Multimodal Summary Generation and Evaluation](https://mlsanity.com/paper/2505.06594/) (2025)
- [Measuring What Matters Beyond Text: Evaluating Multimodal Summaries by Quality, Alignment, and Diversity](https://mlsanity.com/paper/2605.11693/) (2026)

## Abstract (arXiv)

Multimodal summarization aims to generate a concise summary based on the input text and image. However, the existing methods potentially suffer from unfactual output. To evaluate the factuality of multimodal summarization models, we propose two fine-grained and explainable evaluation frameworks (FALLACIOUS) for different application scenarios, i.e. reference-based factuality evaluation framework and reference-free factuality evaluation framework. Notably, the reference-free factuality evaluation framework doesn't need ground truth and hence it has a wider application scenario. To evaluate the effectiveness of the proposed frameworks, we compute the correlation between our frameworks and the other metrics. The experimental results show the effectiveness of our proposed method. We will release our code and dataset via github.