# Modality-Aware Shot Relating and Comparing for Video Scene Detection

arXiv: https://arxiv.org/abs/2412.17238 · HTML: https://mlsanity.com/paper/2412.17238/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 23 Dec 2024.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Bag of Tricks for Image Classification with Convolutional Neural Networks](https://mlsanity.com/paper/1812.01187/) (2018)
- [Graph Convolutional Networks for Text Classification](https://mlsanity.com/paper/1809.05679/) (2018)
- [MovieNet: A Holistic Dataset for Movie Understanding](https://mlsanity.com/paper/2007.10937/) (2020)
- [MovieGraphs: Towards Understanding Human-Centric Situations from Videos](https://mlsanity.com/paper/1712.06761/) (2017)
- [A Local-to-Global Approach to Multi-modal Movie Scene Segmentation](https://mlsanity.com/paper/2004.02678/) (2020)
- [Efficient Movie Scene Detection using State-Space Transformers](https://mlsanity.com/paper/2212.14427/) (2022)
- [Shot Contrastive Self-Supervised Learning for Scene Boundary Detection](https://mlsanity.com/paper/2104.13537/) (2021)
- [A Deep Siamese Network for Scene Detection in Broadcast Videos](https://mlsanity.com/paper/1510.08893/) (2015)
- [Boundary-aware Self-supervised Learning for Video Scene Segmentation](https://mlsanity.com/paper/2201.05277/) (2022)
- [Temporal Perceiver: A General Architecture for Arbitrary Boundary Detection](https://mlsanity.com/paper/2203.00307/) (2022)
- [Movies2Scenes: Using Movie Metadata to Learn Scene Representation](https://mlsanity.com/paper/2202.10650/) (2022)
- [Scene Consistency Representation Learning for Video Scene Segmentation](https://mlsanity.com/paper/2205.05487/) (2022)

## Abstract (arXiv)

Video scene detection involves assessing whether each shot and its surroundings belong to the same scene. Achieving this requires meticulously correlating multi-modal cues, ite.g. visual entity and place modalities, among shots and comparing semantic changes around each shot. However, most methods treat multi-modal semantics equally and do not examine contextual differences between the two sides of a shot, leading to sub-optimal detection performance. In this paper, we propose the bfModality-bfAware bfShot bfRelating and bfComparing approach (MASRC), which enables relating shots per their own characteristics of visual entity and place modalities, as well as comparing multi-shots similarities to have scene changes explicitly encoded. Specifically, to fully harness the potential of visual entity and place modalities in modeling shot relations, we mine long-term shot correlations from entity semantics while simultaneously revealing short-term shot correlations from place semantics. In this way, we can learn distinctive shot features that consolidate coherence within scenes and amplify distinguishability across scenes. Once equipped with distinctive shot features, we further encode the relations between preceding and succeeding shots of each target shot by similarity convolution, aiding in the identification of scene ending shots. We validate the broad applicability of the proposed components in MASRC. Extensive experimental results on public benchmark datasets demonstrate that the proposed MASRC significantly advances video scene detection.