# Causalainer: Causal Explainer for Automatic Video Summarization

arXiv: https://arxiv.org/abs/2305.00455 · HTML: https://mlsanity.com/paper/2305.00455/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 71 references that are in the map.
- First posted on arXiv 30 Apr 2023.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [Advances in Variational Inference](https://mlsanity.com/paper/1711.05597/) (2017)
- [Causal Intervention for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2009.12547/) (2020)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editing](https://mlsanity.com/paper/1912.07538/) (2019)
- [Causal Autoregressive Flows](https://mlsanity.com/paper/2011.02268/) (2020)
- [Video Summarization Using Deep Neural Networks: A Survey](https://mlsanity.com/paper/2101.06072/) (2021)
- [Summarizing Videos with Attention](https://mlsanity.com/paper/1812.01969/) (2018)
- [Theoretical Impediments to Machine Learning With Seven Sparks from the Causal Revolution](https://mlsanity.com/paper/1801.04016/) (2018)
- [Video Summarization with Attention-Based Encoder-Decoder Networks](https://mlsanity.com/paper/1708.09545/) (2017)
- [The Dawn of Quantum Natural Language Processing](https://mlsanity.com/paper/2110.06510/) (2021)
- [Hierarchical Recurrent Neural Network for Video Summarization](https://mlsanity.com/paper/1904.12251/) (2019)
- [DeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanation](https://mlsanity.com/paper/2011.00569/) (2020)
- [GPT2MVS: Generative Pre-trained Transformer-2 for Multi-modal Video Summarization](https://mlsanity.com/paper/2104.12465/) (2021)
- [Query-controllable Video Summarization](https://mlsanity.com/paper/2004.03661/) (2020)
- [Query-adaptive Video Summarization via Quality-aware Relevance Estimation](https://mlsanity.com/paper/1705.00581/) (2017)
- [A Novel Framework for Robustness Analysis of Visual QA Models](https://mlsanity.com/paper/1711.06232/) (2017)
- [Weakly Supervised Video Summarization by Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2001.05864/) (2020)
- [VQABQ: Visual Question Answering by Basic Questions](https://mlsanity.com/paper/1703.06492/) (2017)
- [Robustness Analysis of Visual QA Models by Basic Questions](https://mlsanity.com/paper/1709.04625/) (2017)
- [Assessing the Robustness of Visual Question Answering Models](https://mlsanity.com/paper/1912.01452/) (2019)
- [A Novel Hybrid Machine Learning Model for Auto-Classification of Retinal Diseases](https://mlsanity.com/paper/1806.06423/) (2018)
- [Contextualized Keyword Representations for Multi-modal Retinal Image Captioning](https://mlsanity.com/paper/2104.12471/) (2021)
- [Longer Version for "Deep Context-Encoding Network for Retinal Image Captioning"](https://mlsanity.com/paper/2105.14538/) (2021)
- [Video Summarization using Deep Semantic Features](https://mlsanity.com/paper/1609.08758/) (2016)

## Built on by (most cited first, 8 of 8)

- [Towards Fine-Grained Citation Evaluation in Generated Text: A Comparative Analysis of Faithfulness Metrics](https://mlsanity.com/paper/2406.15264/) (2024)
- [Query-based Video Summarization with Pseudo Label Supervision](https://mlsanity.com/paper/2307.01945/) (2023)
- [QFMTS: Generating Query-Focused Summaries over Multi-Table Inputs](https://mlsanity.com/paper/2405.05109/) (2024)
- [Causal Clothes-Invariant Feature Learning for Cloth-Changing Person Re-ID](https://mlsanity.com/paper/2305.06145/) (2023)
- [Conditional Modeling Based Automatic Video Summarization](https://mlsanity.com/paper/2311.12159/) (2023)
- [Advancing Event Causality Identification via Heuristic Semantic Dependency Inquiry Network](https://mlsanity.com/paper/2409.13621/) (2024)
- [An Integrated Framework for Multi-Granular Explanation of Video Summarization](https://mlsanity.com/paper/2405.10082/) (2024)
- [An Experimental Study on Generating Plausible Textual Explanations for Video Summarization](https://mlsanity.com/paper/2509.26225/) (2025)

## Abstract (arXiv)

The goal of video summarization is to automatically shorten videos such that it conveys the overall story without losing relevant information. In many application scenarios, improper video summarization can have a large impact. For example in forensics, the quality of the generated video summary will affect an investigator's judgment while in journalism it might yield undesired bias. Because of this, modeling explainability is a key concern. One of the best ways to address the explainability challenge is to uncover the causal relations that steer the process and lead to the result. Current machine learning-based video summarization algorithms learn optimal parameters but do not uncover causal relationships. Hence, they suffer from a relative lack of explainability. In this work, a Causal Explainer, dubbed Causalainer, is proposed to address this issue. Multiple meaningful random variables and their joint distributions are introduced to characterize the behaviors of key components in the problem of video summarization. In addition, helper distributions are introduced to enhance the effectiveness of model training. In visual-textual input scenarios, the extra input can decrease the model performance. A causal semantics extractor is designed to tackle this issue by effectively distilling the mutual information from the visual and textual inputs. Experimental results on commonly used benchmarks demonstrate that the proposed method achieves state-of-the-art performance while being more explainable.