# TransVLM: A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions

arXiv: https://arxiv.org/abs/2604.27975 · HTML: https://mlsanity.com/paper/2604.27975/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 51 references that are in the map.
- First posted on arXiv 30 Apr 2026.

## Builds on (most cited first, 28 of 28)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](https://mlsanity.com/paper/2212.03191/) (2022)
- [TransNet V2: An effective deep network architecture for fast shot transition detection](https://mlsanity.com/paper/2008.04838/) (2020)
- [A Local-to-Global Approach to Multi-modal Movie Scene Segmentation](https://mlsanity.com/paper/2004.02678/) (2020)
- [Waver: Wave Your Way to Lifelike Video Generation](https://mlsanity.com/paper/2508.15761/) (2025)
- [VISTA: A Test-Time Self-Improving Video Generation Agent](https://mlsanity.com/paper/2510.15831/) (2025)
- [AutoShot: A Short Video Dataset and State-of-the-Art Shot Boundary Detection](https://mlsanity.com/paper/2304.06116/) (2023)
- [A Deep Siamese Network for Scene Detection in Broadcast Videos](https://mlsanity.com/paper/1510.08893/) (2015)
- [ActionAtlas: A VideoQA Benchmark for Domain-specialized Action Recognition](https://mlsanity.com/paper/2410.05774/) (2024)
- [VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo](https://mlsanity.com/paper/2508.02317/) (2025)
- [Sora Detector: A Unified Hallucination Detection for Large Text-to-Video Models](https://mlsanity.com/paper/2405.04180/) (2024)
- [Fast Video Shot Transition Localization with Deep Structured Models](https://mlsanity.com/paper/1808.04234/) (2018)
- [TransNet: A deep network for fast detection of common shot transitions](https://mlsanity.com/paper/1906.03363/) (2019)
- [NeuFlow: Real-time, High-accuracy Optical Flow Estimation on Robots Using Edge Devices](https://mlsanity.com/paper/2403.10425/) (2024)
- [NeuFlow v2: Push High-Efficiency Optical Flow To the Limit](https://mlsanity.com/paper/2408.10161/) (2024)
- [Enhancing Scene Transition Awareness in Video Generation via Post-Training](https://mlsanity.com/paper/2507.18046/) (2025)

## Built on by (most cited first, 1 of 1)

- [PERSIST: Persistent-State Discrimination for Shot Boundary Detection](https://mlsanity.com/paper/2608.29287/) (2026)

## Abstract (arXiv)

Traditional Shot Boundary Detection (SBD) inherently struggles with complex transitions by formulating the task around isolated cut points, frequently yielding corrupted video shots. We address this fundamental limitation by formalizing the Shot Transition Detection (STD) task. Rather than searching for ambiguous points, STD explicitly detects the continuous temporal segments of transitions. To tackle this, we propose TransVLM, a Vision-Language Model (VLM) framework for STD. Unlike regular VLMs that predominantly rely on spatial semantics and struggle with fine-grained inter-shot dynamics, our method explicitly injects optical flow as a critical motion prior at the input stage. Through a simple yet effective feature-fusion strategy, TransVLM directly processes concatenated color and motion representations, significantly enhancing its temporal awareness without incurring any additional visual token overhead on the language backbone. To overcome the severe class imbalance in public data, we design a scalable data engine to synthesize diverse transition videos for robust training, alongside a comprehensive benchmark for STD. Extensive experiments demonstrate that TransVLM achieves superior overall performance, outperforming traditional heuristic methods, specialized spatiotemporal networks, and top-tier VLMs. For more related research, please visit HeyGen Research (https://www.heygen.com/research) and HeyGen Avatar-V (https://www.heygen.com/research/avatar-v-model). Project page: https://chence17.github.io/TransVLM/