# ERA: Entity Relationship Aware Video Summarization with Wasserstein GAN

arXiv: https://arxiv.org/abs/2109.02625 · HTML: https://mlsanity.com/paper/2109.02625/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 45 references that are in the map.
- First posted on arXiv 6 Sep 2021.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Videos as Space-Time Region Graphs](https://mlsanity.com/paper/1806.01810/) (2018)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [Spatio-Temporal Graph for Video Captioning with Knowledge Distillation](https://mlsanity.com/paper/2003.13942/) (2020)
- [Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward](https://mlsanity.com/paper/1801.00054/) (2017)
- [Video Summarization Using Fully Convolutional Sequence Networks](https://mlsanity.com/paper/1805.10538/) (2018)
- [Video Summarization with Attention-Based Encoder-Decoder Networks](https://mlsanity.com/paper/1708.09545/) (2017)
- [Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning](https://mlsanity.com/paper/1909.02144/) (2019)
- [Discriminative Feature Learning for Unsupervised Video Summarization](https://mlsanity.com/paper/1811.09791/) (2018)
- [Video Summarization by Learning from Unpaired Data](https://mlsanity.com/paper/1805.12174/) (2018)
- [SumGraph: Video Summarization via Recursive Graph Modeling](https://mlsanity.com/paper/2007.08809/) (2020)
- [Learned Video Compression via Joint Spatial-Temporal Correlation Exploration](https://mlsanity.com/paper/1912.06348/) (2019)
- [Spatial-Temporal Correlation and Topology Learning for Person Re-Identification in Videos](https://mlsanity.com/paper/2104.08241/) (2021)
- [Stacked Spatio-Temporal Graph Convolutional Networks for Action Segmentation](https://mlsanity.com/paper/1811.10575/) (2018)
- [Cycle-SUM: Cycle-consistent Adversarial LSTM Networks for Unsupervised Video Summarization](https://mlsanity.com/paper/1904.08265/) (2019)
- [Unsupervised Video Summarization via Multi-source Features](https://mlsanity.com/paper/2105.12532/) (2021)
- [Representation Learning on Visual-Symbolic Graphs for Video Understanding](https://mlsanity.com/paper/1905.07385/) (2019)

## Built on by (most cited first, 3 of 3)

- [VVS: Video-to-Video Retrieval with Irrelevant Frame Suppression](https://mlsanity.com/paper/2303.08906/) (2023)
- [Memory Efficient And Minimax Distribution Estimation Under Wasserstein Distance Using Bayesian Histograms](https://mlsanity.com/paper/2307.10099/) (2023)
- [Language-guided Recursive Spatiotemporal Graph Modeling for Video Summarization](https://mlsanity.com/paper/2509.05604/) (2025)

## Abstract (arXiv)

Video summarization aims to simplify large scale video browsing by generating concise, short summaries that diver from but well represent the original video. Due to the scarcity of video annotations, recent progress for video summarization concentrates on unsupervised methods, among which the GAN based methods are most prevalent. This type of methods includes a summarizer and a discriminator. The summarized video from the summarizer will be assumed as the final output, only if the video reconstructed from this summary cannot be discriminated from the original one by the discriminator. The primary problems of this GAN based methods are two folds. First, the summarized video in this way is a subset of original video with low redundancy and contains high priority events/entities. This summarization criterion is not enough. Second, the training of the GAN framework is not stable. This paper proposes a novel Entity relationship Aware video summarization method (ERA) to address the above problems. To be more specific, we introduce an Adversarial Spatio Temporal network to construct the relationship among entities, which we think should also be given high priority in the summarization. The GAN training problem is solved by introducing the Wasserstein GAN and two newly proposed video patch/score sum losses. In addition, the score sum loss can also relieve the model sensitivity to the varying video lengths, which is an inherent problem for most current video analysis tasks. Our method substantially lifts the performance on the target benchmark datasets and exceeds the current leaderboard Rank 1 state of the art CSNet (2.1% F1 score increase on TVSum and 3.1% F1 score increase on SumMe). We hope our straightforward yet effective approach will shed some light on the future research of unsupervised video summarization.