# A Local-to-Global Approach to Multi-modal Movie Scene Segmentation

arXiv: https://arxiv.org/abs/2004.02678 · HTML: https://mlsanity.com/paper/2004.02678/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Apr 2020.

## Built on by (most cited first, 30 of 60)

- [Long Context Tuning for Video Generation](https://mlsanity.com/paper/2503.10589/) (2025)
- [Category-Level 6D Object Pose Estimation via Cascaded Relation and Recurrent Reconstruction Networks](https://mlsanity.com/paper/2108.08755/) (2021)
- [Efficient Movie Scene Detection using State-Space Transformers](https://mlsanity.com/paper/2212.14427/) (2022)
- [VidChapters-7M: Video Chapters at Scale](https://mlsanity.com/paper/2309.13952/) (2023)
- [Shot Contrastive Self-Supervised Learning for Scene Boundary Detection](https://mlsanity.com/paper/2104.13537/) (2021)
- [Cross-modal Consensus Network for Weakly Supervised Temporal Action Localization](https://mlsanity.com/paper/2107.12589/) (2021)
- [MovieCuts: A New Dataset and Benchmark for Cut Type Recognition](https://mlsanity.com/paper/2109.05569/) (2021)
- [Face, Body, Voice: Video Person-Clustering with Multiple Modalities](https://mlsanity.com/paper/2105.09939/) (2021)
- [AutoShot: A Short Video Dataset and State-of-the-Art Shot Boundary Detection](https://mlsanity.com/paper/2304.06116/) (2023)
- [ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models](https://mlsanity.com/paper/2506.21356/) (2025)
- [VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridges](https://mlsanity.com/paper/2409.01071/) (2024)
- [Learning to Cut by Watching Movies](https://mlsanity.com/paper/2108.04294/) (2021)
- [Cinematographic Camera Diffusion Model](https://mlsanity.com/paper/2402.16143/) (2024)
- [Boundary-aware Self-supervised Learning for Video Scene Segmentation](https://mlsanity.com/paper/2201.05277/) (2022)
- [Generative AI for Film Creation: A Survey of Recent Advances](https://mlsanity.com/paper/2504.08296/) (2025)
- [UnweaveNet: Unweaving Activity Stories](https://mlsanity.com/paper/2112.10194/) (2021)
- [Temporal Perceiver: A General Architecture for Arbitrary Boundary Detection](https://mlsanity.com/paper/2203.00307/) (2022)
- [MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos](https://mlsanity.com/paper/2306.04216/) (2023)
- [Long Story Short: Story-level Video Understanding from 20K Short Films](https://mlsanity.com/paper/2406.10221/) (2024)
- [Movies2Scenes: Using Movie Metadata to Learn Scene Representation](https://mlsanity.com/paper/2202.10650/) (2022)
- [VSTAR: A Video-grounded Dialogue Dataset for Situated Semantic Understanding with Scene and Topic Transitions](https://mlsanity.com/paper/2305.18756/) (2023)
- [Scene Consistency Representation Learning for Video Scene Segmentation](https://mlsanity.com/paper/2205.05487/) (2022)
- [Transformation vs Tradition: Artificial General Intelligence (AGI) for Arts and Humanities](https://mlsanity.com/paper/2310.19626/) (2023)
- [Chapter-Llama: Efficient Chaptering in Hour-Long Videos with LLMs](https://mlsanity.com/paper/2504.00072/) (2025)
- [MHMS: Multimodal Hierarchical Multimedia Summarization](https://mlsanity.com/paper/2204.03734/) (2022)
- [Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment](https://mlsanity.com/paper/2210.04722/) (2022)
- [LiveSeg: Unsupervised Multimodal Temporal Segmentation of Long Livestream Videos](https://mlsanity.com/paper/2210.05840/) (2022)
- [How you feelin'? Learning Emotions and Mental States in Movie Scenes](https://mlsanity.com/paper/2304.05634/) (2023)
- [From Long Videos to Engaging Clips: A Human-Inspired Video Editing Framework with Multimodal Narrative Understanding](https://mlsanity.com/paper/2507.02790/) (2025)
- [DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compression](https://mlsanity.com/paper/2603.15685/) (2026)

## Abstract (arXiv)

Scene, as the crucial unit of storytelling in movies, contains complex activities of actors and their interactions in a physical environment. Identifying the composition of scenes serves as a critical step towards semantic understanding of movies. This is very challenging -- compared to the videos studied in conventional vision problems, e.g. action recognition, as scenes in movies usually contain much richer temporal structures and more complex semantic information. Towards this goal, we scale up the scene segmentation task by building a large-scale video dataset MovieScenes, which contains 21K annotated scene segments from 150 movies. We further propose a local-to-global scene segmentation framework, which integrates multi-modal information across three levels, i.e. clip, segment, and movie. This framework is able to distill complex semantics from hierarchical temporal structures over a long movie, providing top-down guidance for scene segmentation. Our experiments show that the proposed network is able to segment a movie into scenes with high accuracy, consistently outperforming previous methods. We also found that pretraining on our MovieScenes can bring significant improvements to the existing approaches.