# MovieCuts: A New Dataset and Benchmark for Cut Type Recognition

arXiv: https://arxiv.org/abs/2109.05569 · HTML: https://mlsanity.com/paper/2109.05569/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 63 references that are in the map.
- First posted on arXiv 12 Sep 2021.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [MovieQA: Understanding Stories in Movies through Question-Answering](https://mlsanity.com/paper/1512.02902/) (2015)
- [MovieNet: A Holistic Dataset for Movie Understanding](https://mlsanity.com/paper/2007.10937/) (2020)
- [Movie Description](https://mlsanity.com/paper/1605.03705/) (2016)
- [Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets](https://mlsanity.com/paper/2007.09654/) (2020)
- [Condensed Movies: Story Based Retrieval with Contextual Embeddings](https://mlsanity.com/paper/2005.04208/) (2020)
- [MovieGraphs: Towards Understanding Human-Centric Situations from Videos](https://mlsanity.com/paper/1712.06761/) (2017)
- [A Local-to-Global Approach to Multi-modal Movie Scene Segmentation](https://mlsanity.com/paper/2004.02678/) (2020)
- [Multi-shot Temporal Event Localization: a Benchmark](https://mlsanity.com/paper/2012.09434/) (2020)
- [Human Mesh Recovery from Multiple Shots](https://mlsanity.com/paper/2012.09843/) (2020)
- [A Unified Framework for Shot Type Classification Based on Subject Centric Lens](https://mlsanity.com/paper/2008.03548/) (2020)
- [A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering](https://mlsanity.com/paper/1611.07810/) (2016)
- [A Graph-Based Framework to Bridge Movies and Synopses](https://mlsanity.com/paper/1910.11009/) (2019)
- [From Benedict Cumberbatch to Sherlock Holmes: Character Identification in TV series without a Script](https://mlsanity.com/paper/1801.10442/) (2018)
- [Learning to Cut by Watching Movies](https://mlsanity.com/paper/2108.04294/) (2021)
- [Person Search in Videos with One Portrait Through Visual and Temporal Links](https://mlsanity.com/paper/1807.10510/) (2018)
- [Ridiculously Fast Shot Boundary Detection with Fully Convolutional Neural Networks](https://mlsanity.com/paper/1705.08214/) (2017)
- [Playing a Part: Speaker Verification at the Movies](https://mlsanity.com/paper/2010.15716/) (2020)
- [Unifying Identification and Context Learning for Person Recognition](https://mlsanity.com/paper/1806.03084/) (2018)
- [From Trailers to Storylines: An Efficient Way to Learn from Movies](https://mlsanity.com/paper/1806.05341/) (2018)
- [Online Multi-modal Person Search in Videos](https://mlsanity.com/paper/2008.03546/) (2020)
- [Serial Speakers: a Dataset of TV Series](https://mlsanity.com/paper/2002.06923/) (2020)
- [Remembering Winter Was Coming: Character-Oriented Video Summaries of TV Series](https://mlsanity.com/paper/1909.02423/) (2019)

## Built on by (most cited first, 26 of 26)

- [MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptions](https://mlsanity.com/paper/2112.00431/) (2021)
- [AutoAD III: The Prequel -- Back to the Pixels](https://mlsanity.com/paper/2404.14412/) (2024)
- [The Anatomy of Video Editing: A Dataset and Benchmark Suite for AI-Assisted Video Editing](https://mlsanity.com/paper/2207.09812/) (2022)
- [ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models](https://mlsanity.com/paper/2506.21356/) (2025)
- [Dynamic Storyboard Generation in an Engine-based Virtual Environment for Video Production](https://mlsanity.com/paper/2301.12688/) (2023)
- [Cut2Next: Generating Next Shot via In-Context Tuning](https://mlsanity.com/paper/2508.08244/) (2025)
- [Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generation](https://mlsanity.com/paper/2504.01020/) (2025)
- [Finding the Right Moment: Human-Assisted Trailer Creation via Task Composition](https://mlsanity.com/paper/2111.08774/) (2021)
- [Match Cutting: Finding Cuts with Smooth Visual Transitions](https://mlsanity.com/paper/2210.05766/) (2022)
- [Exo2EgoDVC: Dense Video Captioning of Egocentric Procedural Activities Using Web Instructional Videos](https://mlsanity.com/paper/2311.16444/) (2023)
- [Chapter-Llama: Efficient Chaptering in Hour-Long Videos with LLMs](https://mlsanity.com/paper/2504.00072/) (2025)
- [Temporal and Contextual Transformer for Multi-Camera Editing of TV Shows](https://mlsanity.com/paper/2210.08737/) (2022)
- [MM-AU:Towards Multimodal Understanding of Advertisement Videos](https://mlsanity.com/paper/2308.14052/) (2023)
- [Long-range Multimodal Pretraining for Movie Understanding](https://mlsanity.com/paper/2308.09775/) (2023)
- [VEU-Bench: Towards Comprehensive Understanding of Video Editing](https://mlsanity.com/paper/2504.17828/) (2025)
- [MatchDiffusion: Training-free Generation of Match-cuts](https://mlsanity.com/paper/2411.18677/) (2024)
- [Audio visual character profiles for detecting background characters in entertainment media](https://mlsanity.com/paper/2203.11368/) (2022)
- [Audio-Visual Activity Guided Cross-Modal Identity Association for Active Speaker Detection](https://mlsanity.com/paper/2212.00539/) (2022)
- [Visual Objectification in Films: Towards a New AI Task for Video Interpretation](https://mlsanity.com/paper/2401.13296/) (2024)
- [Audio Match Cutting: Finding and Creating Matching Audio Transitions in Movies and Videos](https://mlsanity.com/paper/2408.10998/) (2024)
- [AutoMatch: A Large-scale Audio Beat Matching Benchmark for Boosting Deep Learning Assistant Video Editing](https://mlsanity.com/paper/2303.01884/) (2023)
- [Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understanding](https://mlsanity.com/paper/2508.20765/) (2025)
- [Unsupervised active speaker detection in media content using cross-modal information](https://mlsanity.com/paper/2209.11896/) (2022)
- [V-Trans4Style: Visual Transition Recommendation for Video Production Style Adaptation](https://mlsanity.com/paper/2501.07983/) (2025)
- [MEDit-Bench: A Dataset for Evaluating Message-Driven Narrative Video Editing](https://mlsanity.com/paper/2607.25300/) (2026)
- [Seeking Universal Shot Language Understanding Solutions](https://mlsanity.com/paper/2603.18448/) (2026)

## Abstract (arXiv)

Understanding movies and their structural patterns is a crucial task in decoding the craft of video editing. While previous works have developed tools for general analysis, such as detecting characters or recognizing cinematography properties at the shot level, less effort has been devoted to understanding the most basic video edit, the Cut. This paper introduces the Cut type recognition task, which requires modeling multi-modal information. To ignite research in this new task, we construct a large-scale dataset called MovieCuts, which contains 173,967 video clips labeled with ten cut types defined by professionals in the movie industry. We benchmark a set of audio-visual approaches, including some dealing with the problem's multi-modal nature. Our best model achieves 47.7% mAP, which suggests that the task is challenging and that attaining highly accurate Cut type recognition is an open research problem. Advances in automatic Cut-type recognition can unleash new experiences in the video editing industry, such as movie analysis for education, video re-editing, virtual cinematography, machine-assisted trailer generation, machine-assisted video editing, among others. Our data and code are publicly available: https://github.com/PardoAlejo/MovieCuts}{https://github.com/PardoAlejo/MovieCuts.