# Conditional Modeling Based Automatic Video Summarization

arXiv: https://arxiv.org/abs/2311.12159 · HTML: https://mlsanity.com/paper/2311.12159/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 120 references that are in the map.
- First posted on arXiv 20 Nov 2023.

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [Advances in Variational Inference](https://mlsanity.com/paper/1711.05597/) (2017)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [Identifying Causal Effects With Proxy Variables of an Unmeasured Confounder](https://mlsanity.com/paper/1609.08816/) (2016)
- [Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editing](https://mlsanity.com/paper/1912.07538/) (2019)
- [Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward](https://mlsanity.com/paper/1801.00054/) (2017)
- [Video Summarization Using Deep Neural Networks: A Survey](https://mlsanity.com/paper/2101.06072/) (2021)
- [Video Summarization Using Fully Convolutional Sequence Networks](https://mlsanity.com/paper/1805.10538/) (2018)
- [Summarizing Videos with Attention](https://mlsanity.com/paper/1812.01969/) (2018)
- [Theoretical Impediments to Machine Learning With Seven Sparks from the Causal Revolution](https://mlsanity.com/paper/1801.04016/) (2018)
- [Video Summarization with Attention-Based Encoder-Decoder Networks](https://mlsanity.com/paper/1708.09545/) (2017)
- [The Dawn of Quantum Natural Language Processing](https://mlsanity.com/paper/2110.06510/) (2021)
- [Hierarchical Recurrent Neural Network for Video Summarization](https://mlsanity.com/paper/1904.12251/) (2019)
- [DeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanation](https://mlsanity.com/paper/2011.00569/) (2020)
- [Discriminative Feature Learning for Unsupervised Video Summarization](https://mlsanity.com/paper/1811.09791/) (2018)
- [Video Summarization by Learning from Unpaired Data](https://mlsanity.com/paper/1805.12174/) (2018)
- [GPT2MVS: Generative Pre-trained Transformer-2 for Multi-modal Video Summarization](https://mlsanity.com/paper/2104.12465/) (2021)
- [Cycle-SUM: Cycle-consistent Adversarial LSTM Networks for Unsupervised Video Summarization](https://mlsanity.com/paper/1904.08265/) (2019)
- [Query-controllable Video Summarization](https://mlsanity.com/paper/2004.03661/) (2020)
- [Query-adaptive Video Summarization via Quality-aware Relevance Estimation](https://mlsanity.com/paper/1705.00581/) (2017)
- [A Novel Framework for Robustness Analysis of Visual QA Models](https://mlsanity.com/paper/1711.06232/) (2017)
- [Weakly Supervised Video Summarization by Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2001.05864/) (2020)
- [IntentVizor: Towards Generic Query Guided Interactive Video Summarization](https://mlsanity.com/paper/2109.14834/) (2021)
- [VQABQ: Visual Question Answering by Basic Questions](https://mlsanity.com/paper/1703.06492/) (2017)
- [Collaborative Summarization of Topic-Related Videos](https://mlsanity.com/paper/1706.03114/) (2017)
- [Robustness Analysis of Visual QA Models by Basic Questions](https://mlsanity.com/paper/1709.04625/) (2017)

## Built on by (most cited first, 6 of 6)

- [A Novel Evaluation Framework for Image2Text Generation](https://mlsanity.com/paper/2408.01723/) (2024)
- [QFMTS: Generating Query-Focused Summaries over Multi-Table Inputs](https://mlsanity.com/paper/2405.05109/) (2024)
- [Optimizing Numerical Estimation and Operational Efficiency in the Legal Domain through Large Language Models](https://mlsanity.com/paper/2407.19041/) (2024)
- [Automated Retinal Image Analysis and Medical Report Generation through Deep Learning](https://mlsanity.com/paper/2408.07349/) (2024)
- [Personalized Video Summarization using Text-Based Queries and Conditional Modeling](https://mlsanity.com/paper/2408.14743/) (2024)
- [DeepEyeNet: Generating Medical Report for Retinal Images](https://mlsanity.com/paper/2509.12534/) (2025)

## Abstract (arXiv)

The aim of video summarization is to shorten videos automatically while retaining the key information necessary to convey the overall story. Video summarization methods mainly rely on visual factors, such as visual consecutiveness and diversity, which may not be sufficient to fully understand the content of the video. There are other non-visual factors, such as interestingness, representativeness, and storyline consistency that should also be considered for generating high-quality video summaries. Current methods do not adequately take into account these non-visual factors, resulting in suboptimal performance. In this work, a new approach to video summarization is proposed based on insights gained from how humans create ground truth video summaries. The method utilizes a conditional modeling perspective and introduces multiple meaningful random variables and joint distributions to characterize the key components of video summarization. Helper distributions are employed to improve the training of the model. A conditional attention module is designed to mitigate potential performance degradation in the presence of multi-modal input. The proposed video summarization method incorporates the above innovative design choices that aim to narrow the gap between human-generated and machine-generated video summaries. Extensive experiments show that the proposed approach outperforms existing methods and achieves state-of-the-art performance on commonly used video summarization datasets.