# UniMS: A Unified Framework for Multimodal Summarization with Knowledge Distillation arXiv: https://arxiv.org/abs/2109.05812 ยท HTML: https://mlsanity.com/paper/2109.05812/ ## Key facts - Cited by 23 papers in the map (data through 14 Oct 2026). - Placed in Multimodal distillation, one of 12 directions in Knowledge distillation. - Builds on 16 of its 28 references that are in the map. - First posted on arXiv 13 Sep 2021. ## Placed in - Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/) ## Builds on (most cited first, 16 of 16) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015) - [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019) - [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015) - [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016) - [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017) - [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017) - [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021) - [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017) - [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019) - [Unifying Vision-and-Language Tasks via Text Generation](https://mlsanity.com/paper/2102.02779/) (2021) - [SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents](https://mlsanity.com/paper/1611.04230/) (2016) - [Faithful to the Original: Fact Aware Neural Abstractive Summarization](https://mlsanity.com/paper/1711.04434/) (2017) - [GSum: A General Framework for Guided Neural Abstractive Summarization](https://mlsanity.com/paper/2010.08014/) (2020) ## Built on by (most cited first, 23 of 23) - [UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition](https://mlsanity.com/paper/2211.11256/) (2022) - [UniSA: Unified Generative Framework for Sentiment Analysis](https://mlsanity.com/paper/2309.01339/) (2023) - [Abstractive Text Summarization: State of the Art, Challenges, and Improvements](https://mlsanity.com/paper/2409.02413/) (2024) - [Summary-Oriented Vision Modeling for Multimodal Abstractive Summarization](https://mlsanity.com/paper/2212.07672/) (2022) - [Recent Advances in Multimodal Affective Computing: An NLP Perspective](https://mlsanity.com/paper/2409.07388/) (2024) - [HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks](https://mlsanity.com/paper/2203.03878/) (2022) - [MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos](https://mlsanity.com/paper/2306.04216/) (2023) - [Exploiting Pseudo Image Captions for Multimodal Summarization](https://mlsanity.com/paper/2305.05496/) (2023) - [UniMEEC: Towards Unified Multimodal Emotion Recognition and Emotion Cause](https://mlsanity.com/paper/2404.00403/) (2024) - [Fusing Multimodal Signals on Hyper-complex Space for Extreme Abstractive Text Summarization (TL;DR) of Scientific Contents](https://mlsanity.com/paper/2306.13968/) (2023) - [Evaluating and Improving Factuality in Multimodal Abstractive Summarization](https://mlsanity.com/paper/2211.02580/) (2022) - [Test-Time Distillation for Continual Model Adaptation](https://mlsanity.com/paper/2506.02671/) (2025) - [Leveraging Entity Information for Cross-Modality Correlation Learning: The Entity-Guided Multimodal Summarization](https://mlsanity.com/paper/2408.03149/) (2024) - [i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data](https://mlsanity.com/paper/2305.12311/) (2023) - [D2TV: Dual Knowledge Distillation and Target-oriented Vision Modeling for Many-to-Many Multimodal Summarization](https://mlsanity.com/paper/2305.12767/) (2023) - [MIND: Modality-Informed Knowledge Distillation Framework for Multimodal Clinical Prediction Tasks](https://mlsanity.com/paper/2502.01158/) (2025) - [Modeling Paragraph-Level Vision-Language Semantic Alignment for Multi-Modal Summarization](https://mlsanity.com/paper/2208.11303/) (2022) - [Grafting Pre-trained Models for Multimodal Headline Generation](https://mlsanity.com/paper/2211.07210/) (2022) - [Learning Summary-Worthy Visual Representation for Abstractive Summarization in Video](https://mlsanity.com/paper/2305.04824/) (2023) - [Tell Me What Is Good About This Property: Leveraging Reviews For Segment-Personalized Image Collection Summarization](https://mlsanity.com/paper/2310.19743/) (2023) - [Task Integration Distillation for Object Detectors](https://mlsanity.com/paper/2404.01699/) (2024) - [Self-Regulated Data-Free Knowledge Amalgamation for Text Classification](https://mlsanity.com/paper/2406.15476/) (2024) - [Towards Visually Grounded Multimodal Summarization via Cross-Modal Transformer and Gated Attention](https://mlsanity.com/paper/2605.11753/) (2026) ## Abstract (arXiv) With the rapid increase of multimedia data, a large body of literature has emerged to work on multimodal summarization, the majority of which target at refining salient information from textual and visual modalities to output a pictorial summary with the most relevant images. Existing methods mostly focus on either extractive or abstractive summarization and rely on qualified image captions to build image references. We are the first to propose a Unified framework for Multimodal Summarization grounding on BART, UniMS, that integrates extractive and abstractive objectives, as well as selecting the image output. Specially, we adopt knowledge distillation from a vision-language pretrained model to improve image selection, which avoids any requirement on the existence and quality of image captions. Besides, we introduce a visual guided decoder to better integrate textual and visual modalities in guiding abstractive text generation. Results show that our best model achieves a new state-of-the-art result on a large-scale benchmark dataset. The newly involved extractive objective as well as the knowledge distillation technique are proven to bring a noticeable improvement to the multimodal summarization task.