# Multi-modal Summarization for Video-containing Documents

arXiv: https://arxiv.org/abs/2009.08018 · HTML: https://mlsanity.com/paper/2009.08018/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2020.

## Built on by (most cited first, 13 of 13)

- [Align and Attend: Multimodal Summarization with Dual Contrastive Losses](https://mlsanity.com/paper/2303.07284/) (2023)
- [V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuning](https://mlsanity.com/paper/2404.12353/) (2024)
- [A Survey on Multi-modal Summarization](https://mlsanity.com/paper/2109.05199/) (2021)
- [Self-Supervised Multimodal Opinion Summarization](https://mlsanity.com/paper/2105.13135/) (2021)
- [Hierarchical Cross-Modality Semantic Correlation Learning Model for Multimodal Summarization](https://mlsanity.com/paper/2112.12072/) (2021)
- [Abstractive Text Summarization: State of the Art, Challenges, and Improvements](https://mlsanity.com/paper/2409.02413/) (2024)
- [MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos](https://mlsanity.com/paper/2306.04216/) (2023)
- [Exploiting Pseudo Image Captions for Multimodal Summarization](https://mlsanity.com/paper/2305.05496/) (2023)
- [MHMS: Multimodal Hierarchical Multimedia Summarization](https://mlsanity.com/paper/2204.03734/) (2022)
- [Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment](https://mlsanity.com/paper/2210.04722/) (2022)
- [Grafting Pre-trained Models for Multimodal Headline Generation](https://mlsanity.com/paper/2211.07210/) (2022)
- [UBiSS: A Unified Framework for Bimodal Semantic Summarization of Videos](https://mlsanity.com/paper/2406.16301/) (2024)
- [MS4UI: A Dataset for Multi-modal Summarization of User Interface Instructional Videos](https://mlsanity.com/paper/2506.12623/) (2025)

## Abstract (arXiv)

Summarization of multimedia data becomes increasingly significant as it is the basis for many real-world applications, such as question answering, Web search, and so forth. Most existing multi-modal summarization works however have used visual complementary features extracted from images rather than videos, thereby losing abundant information. Hence, we propose a novel multi-modal summarization task to summarize from a document and its associated video. In this work, we also build a baseline general model with effective strategies, i.e., bi-hop attention and improved late fusion mechanisms to bridge the gap between different modalities, and a bi-stream summarization strategy to employ text and video summarization simultaneously. Comprehensive experiments show that the proposed model is beneficial for multi-modal summarization and superior to existing methods. Moreover, we collect a novel dataset and it provides a new resource for future study that results from documents and videos.