# Reconstructive Sequence-Graph Network for Video Summarization

arXiv: https://arxiv.org/abs/2105.04066 · HTML: https://mlsanity.com/paper/2105.04066/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 53 references that are in the map.
- First posted on arXiv 10 May 2021.

## Builds on (most cited first, 24 of 24)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Graph Convolutional Networks for Temporal Action Localization](https://mlsanity.com/paper/1909.03252/) (2019)
- [Sequence to Sequence -- Video to Text](https://mlsanity.com/paper/1505.00487/) (2015)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [Graph-Based Global Reasoning Networks](https://mlsanity.com/paper/1811.12814/) (2018)
- [Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward](https://mlsanity.com/paper/1801.00054/) (2017)
- [Video Summarization Using Fully Convolutional Sequence Networks](https://mlsanity.com/paper/1805.10538/) (2018)
- [Summarizing Videos with Attention](https://mlsanity.com/paper/1812.01969/) (2018)
- [Rethinking the Evaluation of Video Summaries](https://mlsanity.com/paper/1903.11328/) (2019)
- [Structured Neural Summarization](https://mlsanity.com/paper/1811.01824/) (2018)
- [Hierarchical Recurrent Neural Network for Video Summarization](https://mlsanity.com/paper/1904.12251/) (2019)
- [Query-Focused Video Summarization: Dataset, Evaluation, and A Memory Network Based Approach](https://mlsanity.com/paper/1707.04960/) (2017)
- [Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning](https://mlsanity.com/paper/1511.03476/) (2015)
- [Summary Transfer: Exemplar-based Subset Selection for Video Summarization](https://mlsanity.com/paper/1603.03369/) (2016)
- [Title Generation for User Generated Videos](https://mlsanity.com/paper/1608.07068/) (2016)
- [SocialGCN: An Efficient Graph Convolutional Network based Model for Social Recommendation](https://mlsanity.com/paper/1811.02815/) (2018)
- [Convolutional Hierarchical Attention Network for Query-Focused Video Summarization](https://mlsanity.com/paper/2002.03740/) (2020)
- [A General Framework for Edited Video and Raw Video Summarization](https://mlsanity.com/paper/1904.10669/) (2019)
- [Hierarchical Video Frame Sequence Representation with Deep Convolutional Graph Network](https://mlsanity.com/paper/1906.00377/) (2019)
- [Attention is all you need for Videos: Self-attention based Video Summarization using Universal Transformers](https://mlsanity.com/paper/1906.02792/) (2019)
- [ILS-SUMM: Iterated Local Search for Unsupervised Video Summarization](https://mlsanity.com/paper/1912.03650/) (2019)

## Built on by (most cited first, 30 of 34)

- [Align and Attend: Multimodal Summarization with Dual Contrastive Losses](https://mlsanity.com/paper/2303.07284/) (2023)
- [AI Flow: Perspectives, Scenarios, and Approaches](https://mlsanity.com/paper/2506.12479/) (2025)
- [Progressive Video Summarization via Multimodal Self-supervised Learning](https://mlsanity.com/paper/2201.02494/) (2022)
- [Scaling Up Video Summarization Pretraining with Large Language Models](https://mlsanity.com/paper/2404.03398/) (2024)
- [Learning Pixel-Level Distinctions for Video Highlight Detection](https://mlsanity.com/paper/2204.04615/) (2022)
- [Video Summarization with Large Language Models](https://mlsanity.com/paper/2504.11199/) (2025)
- [Audio-visual Representation Learning for Anomaly Events Detection in Crowds](https://mlsanity.com/paper/2110.14862/) (2021)
- [Multimodal Subtask Graph Generation from Instructional Videos](https://mlsanity.com/paper/2302.08672/) (2023)
- [MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos](https://mlsanity.com/paper/2306.04216/) (2023)
- [Congested Crowd Instance Localization with Dilated Convolutional Swin Transformer](https://mlsanity.com/paper/2108.00584/) (2021)
- [Learning a Condensed Frame for Memory-Efficient Video Class-Incremental Learning](https://mlsanity.com/paper/2211.00833/) (2022)
- [Video Joint Modelling Based on Hierarchical Transformer for Co-summarization](https://mlsanity.com/paper/2112.13478/) (2021)
- [EA-Net: Edge-Aware Network for Flow-based Video Frame Interpolation](https://mlsanity.com/paper/2105.07673/) (2021)
- [VVS: Video-to-Video Retrieval with Irrelevant Frame Suppression](https://mlsanity.com/paper/2303.08906/) (2023)
- [Spiking Variational Graph Representation Inference for Video Summarization](https://mlsanity.com/paper/2508.15389/) (2025)
- [Multimodal Frame-Scoring Transformer for Video Summarization](https://mlsanity.com/paper/2207.01814/) (2022)
- [VideoSAGE: Video Summarization with Graph Representation Learning](https://mlsanity.com/paper/2404.10539/) (2024)
- [Language-Guided Self-Supervised Video Summarization Using Text Semantic Matching Considering the Diversity of the Video](https://mlsanity.com/paper/2405.08890/) (2024)
- [Low-Light Hyperspectral Image Enhancement](https://mlsanity.com/paper/2208.03042/) (2022)
- [TripleSumm: Adaptive Triple-Modality Fusion for Video Summarization](https://mlsanity.com/paper/2603.01169/) (2026)
- [Working memory inspired hierarchical video decomposition with transformative representations](https://mlsanity.com/paper/2204.10105/) (2022)
- [Cross-modal Generative Model for Visual-Guided Binaural Stereo Generation](https://mlsanity.com/paper/2311.07630/) (2023)
- [Facilitating the Production of Well-tailored Video Summaries for Sharing on Social Media](https://mlsanity.com/paper/2312.02616/) (2023)
- [Video Summarization using Denoising Diffusion Probabilistic Model](https://mlsanity.com/paper/2412.08357/) (2024)
- [Early Exit and Multi Stage Knowledge Distillation in VLMs for Video Summarization](https://mlsanity.com/paper/2504.21831/) (2025)
- [MEDit-Bench: A Dataset for Evaluating Message-Driven Narrative Video Editing](https://mlsanity.com/paper/2607.25300/) (2026)
- [MAFNet: A Multi-Attention Fusion Network for RGB-T Crowd Counting](https://mlsanity.com/paper/2208.06761/) (2022)
- [SELF-VS: Self-supervised Encoding Learning For Video Summarization](https://mlsanity.com/paper/2303.15993/) (2023)
- [Does SpatioTemporal information benefit Two video summarization benchmarks?](https://mlsanity.com/paper/2410.03323/) (2024)
- [FullTransNet: Full Transformer with Local-Global Attention for Video Summarization](https://mlsanity.com/paper/2501.00882/) (2025)

## Abstract (arXiv)

Exploiting the inner-shot and inter-shot dependencies is essential for key-shot based video summarization. Current approaches mainly devote to modeling the video as a frame sequence by recurrent neural networks. However, one potential limitation of the sequence models is that they focus on capturing local neighborhood dependencies while the high-order dependencies in long distance are not fully exploited. In general, the frames in each shot record a certain activity and vary smoothly over time, but the multi-hop relationships occur frequently among shots. In this case, both the local and global dependencies are important for understanding the video content. Motivated by this point, we propose a Reconstructive Sequence-Graph Network (RSGN) to encode the frames and shots as sequence and graph hierarchically, where the frame-level dependencies are encoded by Long Short-Term Memory (LSTM), and the shot-level dependencies are captured by the Graph Convolutional Network (GCN). Then, the videos are summarized by exploiting both the local and global dependencies among shots. Besides, a reconstructor is developed to reward the summary generator, so that the generator can be optimized in an unsupervised manner, which can avert the lack of annotated data in video summarization. Furthermore, under the guidance of reconstruction loss, the predicted summary can better preserve the main video content and shot-level dependencies. Practically, the experimental results on three popular datasets i.e., SumMe, TVsum and VTW) have demonstrated the superiority of our proposed approach to the summarization task.