# Scene-VLM: Multimodal Video Scene Segmentation via Vision-Language Models

arXiv: https://arxiv.org/abs/2512.21778 · HTML: https://mlsanity.com/paper/2512.21778/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 25 Dec 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 21 of 21)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [MovieNet: A Holistic Dataset for Movie Understanding](https://mlsanity.com/paper/2007.10937/) (2020)
- [A Local-to-Global Approach to Multi-modal Movie Scene Segmentation](https://mlsanity.com/paper/2004.02678/) (2020)
- [Efficient Movie Scene Detection using State-Space Transformers](https://mlsanity.com/paper/2212.14427/) (2022)
- [VidChapters-7M: Video Chapters at Scale](https://mlsanity.com/paper/2309.13952/) (2023)
- [Shot Contrastive Self-Supervised Learning for Scene Boundary Detection](https://mlsanity.com/paper/2104.13537/) (2021)
- [A Deep Siamese Network for Scene Detection in Broadcast Videos](https://mlsanity.com/paper/1510.08893/) (2015)
- [Boundary-aware Self-supervised Learning for Video Scene Segmentation](https://mlsanity.com/paper/2201.05277/) (2022)
- [Scene Consistency Representation Learning for Video Scene Segmentation](https://mlsanity.com/paper/2205.05487/) (2022)
- [Chapter-Llama: Efficient Chaptering in Hour-Long Videos with LLMs](https://mlsanity.com/paper/2504.00072/) (2025)
- [MEGA: Multimodal Alignment Aggregation and Distillation For Cinematic Video Segmentation](https://mlsanity.com/paper/2308.11185/) (2023)
- [LV-MAE: Learning Long Video Representations through Masked-Embedding Autoencoders](https://mlsanity.com/paper/2504.03501/) (2025)

## Built on by (most cited first, 2 of 2)

- [OmniShotCut: Holistic Relational Shot Boundary Detection with Shot-Query Transformer](https://mlsanity.com/paper/2604.24762/) (2026)
- [Beyond Visual Boundaries: Rethinking Scene Segmentation for Movie RAG](https://mlsanity.com/paper/2608.28699/) (2026)

## Abstract (arXiv)

Segmenting long-form videos into semantically coherent scenes is a fundamental task in large-scale video understanding. Existing encoder-based methods are limited by visual-centric biases, classify each shot in isolation without leveraging sequential dependencies, and lack both narrative understanding and explainability. In this paper, we present Scene-VLM, the first fine-tuned vision-language model (VLM) framework for video scene segmentation. Scene-VLM jointly processes visual and textual cues including frames, transcriptions, and optional metadata to enable multimodal reasoning across consecutive shots. The model generates predictions sequentially with causal dependencies among shots and introduces a context-focus window mechanism to ensure sufficient temporal context for each shot-level decision. In addition, we propose a scheme to extract confidence scores from the token-level logits of the VLM, enabling controllable precision-recall trade-offs that were previously limited to encoder-based methods. Furthermore, we demonstrate that our model can be aligned to generate coherent natural-language rationales for its boundary decisions through minimal targeted supervision. Our approach achieves state-of-the-art performance on standard scene segmentation benchmarks. On MovieNet, for example, Scene-VLM yields significant improvements of +6 AP and +13.7 F1 over the previous leading method.