# Causal Video Summarizer for Video Exploration

arXiv: https://arxiv.org/abs/2307.01947 · HTML: https://mlsanity.com/paper/2307.01947/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 51 references that are in the map.
- First posted on arXiv 4 Jul 2023.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [Causal Intervention for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2009.12547/) (2020)
- [Identifying Causal Effects With Proxy Variables of an Unmeasured Confounder](https://mlsanity.com/paper/1609.08816/) (2016)
- [Video Summarization Using Deep Neural Networks: A Survey](https://mlsanity.com/paper/2101.06072/) (2021)
- [Video Summarization with Attention-Based Encoder-Decoder Networks](https://mlsanity.com/paper/1708.09545/) (2017)
- [The Dawn of Quantum Natural Language Processing](https://mlsanity.com/paper/2110.06510/) (2021)
- [DeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanation](https://mlsanity.com/paper/2011.00569/) (2020)
- [GPT2MVS: Generative Pre-trained Transformer-2 for Multi-modal Video Summarization](https://mlsanity.com/paper/2104.12465/) (2021)
- [Summary Transfer: Exemplar-based Subset Selection for Video Summarization](https://mlsanity.com/paper/1603.03369/) (2016)
- [Query-controllable Video Summarization](https://mlsanity.com/paper/2004.03661/) (2020)
- [Query-adaptive Video Summarization via Quality-aware Relevance Estimation](https://mlsanity.com/paper/1705.00581/) (2017)
- [A Novel Framework for Robustness Analysis of Visual QA Models](https://mlsanity.com/paper/1711.06232/) (2017)
- [VQABQ: Visual Question Answering by Basic Questions](https://mlsanity.com/paper/1703.06492/) (2017)
- [Robustness Analysis of Visual QA Models by Basic Questions](https://mlsanity.com/paper/1709.04625/) (2017)
- [Assessing the Robustness of Visual Question Answering Models](https://mlsanity.com/paper/1912.01452/) (2019)
- [A Novel Hybrid Machine Learning Model for Auto-Classification of Retinal Diseases](https://mlsanity.com/paper/1806.06423/) (2018)
- [Contextualized Keyword Representations for Multi-modal Retinal Image Captioning](https://mlsanity.com/paper/2104.12471/) (2021)
- [Longer Version for "Deep Context-Encoding Network for Retinal Image Captioning"](https://mlsanity.com/paper/2105.14538/) (2021)
- [Synthesizing New Retinal Symptom Images by Multiple Generative Models](https://mlsanity.com/paper/1902.04147/) (2019)
- [Dilated Temporal Relational Adversarial Network for Generic Video Summarization](https://mlsanity.com/paper/1804.11228/) (2018)
- [Improving Sequential Determinantal Point Processes for Supervised Video Summarization](https://mlsanity.com/paper/1807.10957/) (2018)
- [Training a Resilient Q-Network against Observational Interference](https://mlsanity.com/paper/2102.09677/) (2021)

## Built on by (most cited first, 4 of 4)

- [Towards Fine-Grained Citation Evaluation in Generated Text: A Comparative Analysis of Faithfulness Metrics](https://mlsanity.com/paper/2406.15264/) (2024)
- [QFMTS: Generating Query-Focused Summaries over Multi-Table Inputs](https://mlsanity.com/paper/2405.05109/) (2024)
- [Conditional Modeling Based Automatic Video Summarization](https://mlsanity.com/paper/2311.12159/) (2023)
- [A Novel Automatic Framework for Speaker Drift Detection in Synthesized Speech](https://mlsanity.com/paper/2604.06327/) (2026)

## Abstract (arXiv)

Recently, video summarization has been proposed as a method to help video exploration. However, traditional video summarization models only generate a fixed video summary which is usually independent of user-specific needs and hence limits the effectiveness of video exploration. Multi-modal video summarization is one of the approaches utilized to address this issue. Multi-modal video summarization has a video input and a text-based query input. Hence, effective modeling of the interaction between a video input and text-based query is essential to multi-modal video summarization. In this work, a new causality-based method named Causal Video Summarizer (CVS) is proposed to effectively capture the interactive information between the video and query to tackle the task of multi-modal video summarization. The proposed method consists of a probabilistic encoder and a probabilistic decoder. Based on the evaluation of the existing multi-modal video summarization dataset, experimental results show that the proposed approach is effective with the increase of +5.4% in accuracy and +4.92% increase of F 1- score, compared with the state-of-the-art method.