# MAST: Multimodal Abstractive Summarization with Trimodal Hierarchical Attention

arXiv: https://arxiv.org/abs/2010.08021 · HTML: https://mlsanity.com/paper/2010.08021/

## Key facts

- Cited by 12 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 15 Oct 2020.

## Built on by (most cited first, 12 of 12)

- [Align and Attend: Multimodal Summarization with Dual Contrastive Losses](https://mlsanity.com/paper/2303.07284/) (2023)
- [Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization](https://mlsanity.com/paper/2109.02401/) (2021)
- [A Survey on Multi-modal Summarization](https://mlsanity.com/paper/2109.05199/) (2021)
- [Hierarchical Cross-Modality Semantic Correlation Learning Model for Multimodal Summarization](https://mlsanity.com/paper/2112.12072/) (2021)
- [Abstractive Text Summarization: State of the Art, Challenges, and Improvements](https://mlsanity.com/paper/2409.02413/) (2024)
- [MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos](https://mlsanity.com/paper/2306.04216/) (2023)
- [MetaSumPerceiver: Multimodal Multi-Document Evidence Summarization for Fact-Checking](https://mlsanity.com/paper/2407.13089/) (2024)
- [Summarizing Speech: A Comprehensive Survey](https://mlsanity.com/paper/2504.08024/) (2025)
- [CISum: Learning Cross-modality Interaction to Enhance Multimodal Semantic Coverage for Multimodal Summarization](https://mlsanity.com/paper/2302.09934/) (2023)
- [Modeling Paragraph-Level Vision-Language Semantic Alignment for Multi-Modal Summarization](https://mlsanity.com/paper/2208.11303/) (2022)
- [Grafting Pre-trained Models for Multimodal Headline Generation](https://mlsanity.com/paper/2211.07210/) (2022)
- [TCR: Short Video Title Generation and Cover Selection with Attention Refinement](https://mlsanity.com/paper/2304.12561/) (2023)

## Abstract (arXiv)

This paper presents MAST, a new model for Multimodal Abstractive Text Summarization that utilizes information from all three modalities -- text, audio and video -- in a multimodal video. Prior work on multimodal abstractive text summarization only utilized information from the text and video modalities. We examine the usefulness and challenges of deriving information from the audio modality and present a sequence-to-sequence trimodal hierarchical attention-based model that overcomes these challenges by letting the model pay more attention to the text modality. MAST outperforms the current state of the art model (video-text) by 2.51 points in terms of Content F1 score and 1.00 points in terms of Rouge-L score on the How2 dataset for multimodal language understanding.