# GPT2MVS: Generative Pre-trained Transformer-2 for Multi-modal Video Summarization

arXiv: https://arxiv.org/abs/2104.12465 · HTML: https://mlsanity.com/paper/2104.12465/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 74 references that are in the map.
- First posted on arXiv 26 Apr 2021.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward](https://mlsanity.com/paper/1801.00054/) (2017)
- [Video Summarization Using Deep Neural Networks: A Survey](https://mlsanity.com/paper/2101.06072/) (2021)
- [Video Summarization with Attention-Based Encoder-Decoder Networks](https://mlsanity.com/paper/1708.09545/) (2017)
- [Hierarchical Recurrent Neural Network for Video Summarization](https://mlsanity.com/paper/1904.12251/) (2019)
- [DeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanation](https://mlsanity.com/paper/2011.00569/) (2020)
- [Discriminative Feature Learning for Unsupervised Video Summarization](https://mlsanity.com/paper/1811.09791/) (2018)
- [Video Summarization by Learning from Unpaired Data](https://mlsanity.com/paper/1805.12174/) (2018)
- [Summary Transfer: Exemplar-based Subset Selection for Video Summarization](https://mlsanity.com/paper/1603.03369/) (2016)
- [Cycle-SUM: Cycle-consistent Adversarial LSTM Networks for Unsupervised Video Summarization](https://mlsanity.com/paper/1904.08265/) (2019)
- [Query-controllable Video Summarization](https://mlsanity.com/paper/2004.03661/) (2020)
- [Query-adaptive Video Summarization via Quality-aware Relevance Estimation](https://mlsanity.com/paper/1705.00581/) (2017)
- [A Novel Framework for Robustness Analysis of Visual QA Models](https://mlsanity.com/paper/1711.06232/) (2017)
- [Weakly Supervised Video Summarization by Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2001.05864/) (2020)
- [VQABQ: Visual Question Answering by Basic Questions](https://mlsanity.com/paper/1703.06492/) (2017)
- [Collaborative Summarization of Topic-Related Videos](https://mlsanity.com/paper/1706.03114/) (2017)
- [Robustness Analysis of Visual QA Models by Basic Questions](https://mlsanity.com/paper/1709.04625/) (2017)
- [Assessing the Robustness of Visual Question Answering Models](https://mlsanity.com/paper/1912.01452/) (2019)
- [A Novel Hybrid Machine Learning Model for Auto-Classification of Retinal Diseases](https://mlsanity.com/paper/1806.06423/) (2018)
- [Video Summarization using Deep Semantic Features](https://mlsanity.com/paper/1609.08758/) (2016)

## Built on by (most cited first, 26 of 26)

- [The Dawn of Quantum Natural Language Processing](https://mlsanity.com/paper/2110.06510/) (2021)
- [V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuning](https://mlsanity.com/paper/2404.12353/) (2024)
- [Towards Fine-Grained Citation Evaluation in Generated Text: A Comparative Analysis of Faithfulness Metrics](https://mlsanity.com/paper/2406.15264/) (2024)
- [Longer Version for "Deep Context-Encoding Network for Retinal Image Captioning"](https://mlsanity.com/paper/2105.14538/) (2021)
- [Backbones-Review: Feature Extraction Networks for Deep Learning and Deep Reinforcement Learning Approaches](https://mlsanity.com/paper/2206.08016/) (2022)
- [Query-based Video Summarization with Pseudo Label Supervision](https://mlsanity.com/paper/2307.01945/) (2023)
- [A Novel Evaluation Framework for Image2Text Generation](https://mlsanity.com/paper/2408.01723/) (2024)
- [Improving Visual Question Answering Models through Robustness Analysis and In-Context Learning with a Chain of Basic Questions](https://mlsanity.com/paper/2304.03147/) (2023)
- [QFMTS: Generating Query-Focused Summaries over Multi-Table Inputs](https://mlsanity.com/paper/2405.05109/) (2024)
- [Causalainer: Causal Explainer for Automatic Video Summarization](https://mlsanity.com/paper/2305.00455/) (2023)
- [Experience and Evidence are the eyes of an excellent summarizer! Towards Knowledge Infused Multi-modal Clinical Conversation Summarization](https://mlsanity.com/paper/2309.15739/) (2023)
- [Conditional Modeling Based Automatic Video Summarization](https://mlsanity.com/paper/2311.12159/) (2023)
- [Optimizing Numerical Estimation and Operational Efficiency in the Legal Domain through Large Language Models](https://mlsanity.com/paper/2407.19041/) (2024)
- [Causal Video Summarizer for Video Exploration](https://mlsanity.com/paper/2307.01947/) (2023)
- [Reviewing Intelligent Cinematography: AI research for camera-based video production](https://mlsanity.com/paper/2405.05039/) (2024)
- [SummDiff: Generative Modeling of Video Summarization with Diffusion](https://mlsanity.com/paper/2510.08458/) (2025)
- [MHSCNet: A Multimodal Hierarchical Shot-aware Convolutional Network for Video Summarization](https://mlsanity.com/paper/2204.08352/) (2022)
- [A Comparative Analysis of Faithfulness Metrics and Humans in Citation Evaluation](https://mlsanity.com/paper/2408.12398/) (2024)
- [Video Summarisation with Incident and Context Information using Generative AI](https://mlsanity.com/paper/2501.04764/) (2025)
- [TripleSumm: Adaptive Triple-Modality Fusion for Video Summarization](https://mlsanity.com/paper/2603.01169/) (2026)
- [Robust Unsupervised Multi-Object Tracking in Noisy Environments](https://mlsanity.com/paper/2105.10005/) (2021)
- [DeepEyeNet: Generating Medical Report for Retinal Images](https://mlsanity.com/paper/2509.12534/) (2025)
- [Semantic Frame Aggregation-based Transformer for Live Video Comment Generation](https://mlsanity.com/paper/2510.26978/) (2025)
- [Yes, this is what I was looking for! Towards Multi-modal Medical Consultation Concern Summary Generation](https://mlsanity.com/paper/2401.05134/) (2024)
- [Find the Cliffhanger: Multi-Modal Trailerness in Soap Operas](https://mlsanity.com/paper/2401.16076/) (2024)
- [A Novel Automatic Framework for Speaker Drift Detection in Synthesized Speech](https://mlsanity.com/paper/2604.06327/) (2026)

## Abstract (arXiv)

Traditional video summarization methods generate fixed video representations regardless of user interest. Therefore such methods limit users' expectations in content search and exploration scenarios. Multi-modal video summarization is one of the methods utilized to address this problem. When multi-modal video summarization is used to help video exploration, a text-based query is considered as one of the main drivers of video summary generation, as it is user-defined. Thus, encoding the text-based query and the video effectively are both important for the task of multi-modal video summarization. In this work, a new method is proposed that uses a specialized attention network and contextualized word representations to tackle this task. The proposed model consists of a contextualized video summary controller, multi-modal attention mechanisms, an interactive attention network, and a video summary generator. Based on the evaluation of the existing multi-modal video summarization benchmark, experimental results show that the proposed model is effective with the increase of +5.88% in accuracy and +4.06% increase of F1-score, compared with the state-of-the-art method.