# CSTA: CNN-based Spatiotemporal Attention for Video Summarization

arXiv: https://arxiv.org/abs/2405.11905 · HTML: https://mlsanity.com/paper/2405.11905/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 49 references that are in the map.
- First posted on arXiv 20 May 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [CMT: Convolutional Neural Networks Meet Vision Transformers](https://mlsanity.com/paper/2107.06263/) (2021)
- [Incorporating Convolution Designs into Visual Transformers](https://mlsanity.com/paper/2103.11816/) (2021)
- [How Much Position Information Do Convolutional Neural Networks Encode?](https://mlsanity.com/paper/2001.08248/) (2020)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [On Translation Invariance in CNNs: Convolutional Layers can Exploit Absolute Spatial Location](https://mlsanity.com/paper/2003.07064/) (2020)
- [CLIP-It! Language-Guided Video Summarization](https://mlsanity.com/paper/2107.00650/) (2021)
- [Summarizing Videos with Attention](https://mlsanity.com/paper/1812.01969/) (2018)
- [Align and Attend: Multimodal Summarization with Dual Contrastive Losses](https://mlsanity.com/paper/2303.07284/) (2023)
- [Video Summarization with Attention-Based Encoder-Decoder Networks](https://mlsanity.com/paper/1708.09545/) (2017)
- [Rethinking the Evaluation of Video Summaries](https://mlsanity.com/paper/1903.11328/) (2019)
- [Hierarchical Recurrent Neural Network for Video Summarization](https://mlsanity.com/paper/1904.12251/) (2019)
- [Exploring global diverse attention via pairwise temporal relation for video summarization](https://mlsanity.com/paper/2009.10942/) (2020)
- [Supervised Video Summarization via Multiple Feature Sets with Parallel Attention](https://mlsanity.com/paper/2104.11530/) (2021)
- [Progressive Video Summarization via Multimodal Self-supervised Learning](https://mlsanity.com/paper/2201.02494/) (2022)
- [Hierarchical Multimodal Transformer to Summarize Videos](https://mlsanity.com/paper/2109.10559/) (2021)
- [Query Twice: Dual Mixture Attention Meta Learning for Video Summarization](https://mlsanity.com/paper/2008.08360/) (2020)
- [Video Joint Modelling Based on Hierarchical Transformer for Co-summarization](https://mlsanity.com/paper/2112.13478/) (2021)

## Built on by (most cited first, 23 of 23)

- [VidHal: Benchmarking Temporal Hallucinations in Vision LLMs](https://mlsanity.com/paper/2411.16771/) (2024)
- [Video2BEV: Transforming Drone Videos to BEVs for Video-based Geo-localization](https://mlsanity.com/paper/2411.13610/) (2024)
- [Video Summarization with Large Language Models](https://mlsanity.com/paper/2504.11199/) (2025)
- [LVMark: Robust Watermark for Latent Video Diffusion Models](https://mlsanity.com/paper/2412.09122/) (2024)
- [Frame Sampling Strategies Matter: A Benchmark for small vision language models](https://mlsanity.com/paper/2509.14769/) (2025)
- [SummDiff: Generative Modeling of Video Summarization with Diffusion](https://mlsanity.com/paper/2510.08458/) (2025)
- [MS-Temba: Multi-Scale Temporal Mamba for Understanding Long Untrimmed Videos](https://mlsanity.com/paper/2501.06138/) (2025)
- [TRIM: A Self-Supervised Video Summarization Framework Maximizing Temporal Relative Information and Representativeness](https://mlsanity.com/paper/2506.20588/) (2025)
- [TripleSumm: Adaptive Triple-Modality Fusion for Video Summarization](https://mlsanity.com/paper/2603.01169/) (2026)
- [PEEK: Picking Essential frames via Efficient Knowledge distillation](https://mlsanity.com/paper/2605.31029/) (2026)
- [Sounding Highlights: Dual-Pathway Audio Encoders for Audio-Visual Video Highlight Detection](https://mlsanity.com/paper/2602.03891/) (2026)
- [TRIMMER: A New Paradigm for Video Summarization through Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/2605.01659/) (2026)
- [MEDit-Bench: A Dataset for Evaluating Message-Driven Narrative Video Editing](https://mlsanity.com/paper/2607.25300/) (2026)
- [Graph Unfolding and Sampling for Transitory Video Keyframe Selection via Gershgorin Disc Alignment](https://mlsanity.com/paper/2408.01859/) (2024)
- [Video Dataset Condensation with Diffusion Models](https://mlsanity.com/paper/2505.06670/) (2025)
- [SD-MVSum: Script-Driven Multimodal Video Summarization Method and Datasets](https://mlsanity.com/paper/2510.05652/) (2025)
- [AdSum: Two-stream Audio-visual Summarization for Automated Video Advertisement Clipping](https://mlsanity.com/paper/2510.26569/) (2025)
- [Uncertainty-Aware and Decoder-Aligned Learning for Video Summarization](https://mlsanity.com/paper/2605.09507/) (2026)
- [HAS: Highlight-guided Attention Steering for Multimodal LLM Video Summarization](https://mlsanity.com/paper/2607.17994/) (2026)
- [FORGE: Frame Orthogonality in Relevance Geometry for Long-Form Video Understanding](https://mlsanity.com/paper/2607.25266/) (2026)
- [TRINITY: A Multi-Perspective Benchmark for Personal-Style Video Highlight Detection](https://mlsanity.com/paper/2608.29577/) (2026)
- [SGWIB:Sliced Gromov-Wasserstein Information Bottleneck for Video Highlight Detection](https://mlsanity.com/paper/2609.13966/) (2026)
- [TSMD: Temporal-Stream Modality Dropout for Robust Video Highlight Detection](https://mlsanity.com/paper/2609.39051/) (2026)

## Abstract (arXiv)

Video summarization aims to generate a concise representation of a video, capturing its essential content and key moments while reducing its overall length. Although several methods employ attention mechanisms to handle long-term dependencies, they often fail to capture the visual significance inherent in frames. To address this limitation, we propose a CNN-based SpatioTemporal Attention (CSTA) method that stacks each feature of frames from a single video to form image-like frame representations and applies 2D CNN to these frame features. Our methodology relies on CNN to comprehend the inter and intra-frame relations and to find crucial attributes in videos by exploiting its ability to learn absolute positions within images. In contrast to previous work compromising efficiency by designing additional modules to focus on spatial importance, CSTA requires minimal computational overhead as it uses CNN as a sliding window. Extensive experiments on two benchmark datasets (SumMe and TVSum) demonstrate that our proposed approach achieves state-of-the-art performance with fewer MACs compared to previous methods. Codes are available at https://github.com/thswodnjs3/CSTA.