# Learning to Cut by Watching Movies

arXiv: https://arxiv.org/abs/2108.04294 · HTML: https://mlsanity.com/paper/2108.04294/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 72 references that are in the map.
- First posted on arXiv 9 Aug 2021.

## Builds on (most cited first, 30 of 52)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [End-to-End Learning of Visual Representations from Uncurated Instructional Videos](https://mlsanity.com/paper/1912.06430/) (2019)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [Hard Negative Mixing for Contrastive Learning](https://mlsanity.com/paper/2010.01028/) (2020)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training](https://mlsanity.com/paper/2005.00200/) (2020)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [Self-Supervised MultiModal Versatile Networks](https://mlsanity.com/paper/2006.16228/) (2020)
- [Self-Supervised Learning by Cross-Modal Audio-Video Clustering](https://mlsanity.com/paper/1911.12667/) (2019)
- [Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization](https://mlsanity.com/paper/1807.00230/) (2018)
- [Self-supervised Co-training for Video Representation Learning](https://mlsanity.com/paper/2010.09709/) (2020)
- [TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval](https://mlsanity.com/paper/2001.09099/) (2020)
- [Use What You Have: Video Retrieval Using Representations From Collaborative Experts](https://mlsanity.com/paper/1907.13487/) (2019)
- [MovieNet: A Holistic Dataset for Movie Understanding](https://mlsanity.com/paper/2007.10937/) (2020)
- [Self-Supervised Learning of Audio-Visual Objects from Video](https://mlsanity.com/paper/2008.04237/) (2020)
- [Video Representation Learning by Dense Predictive Coding](https://mlsanity.com/paper/1909.04656/) (2019)
- [Text-based Editing of Talking-head Video](https://mlsanity.com/paper/1906.01524/) (2019)
- [Condensed Movies: Story Based Retrieval with Contextual Embeddings](https://mlsanity.com/paper/2005.04208/) (2020)
- [Learning Representations from Audio-Visual Spatial Alignment](https://mlsanity.com/paper/2011.01819/) (2020)

## Built on by (most cited first, 18 of 18)

- [MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptions](https://mlsanity.com/paper/2112.00431/) (2021)
- [MovieCuts: A New Dataset and Benchmark for Cut Type Recognition](https://mlsanity.com/paper/2109.05569/) (2021)
- [The Anatomy of Video Editing: A Dataset and Benchmark Suite for AI-Assisted Video Editing](https://mlsanity.com/paper/2207.09812/) (2022)
- [Dynamic Storyboard Generation in an Engine-based Virtual Environment for Video Production](https://mlsanity.com/paper/2301.12688/) (2023)
- [Cinematographic Camera Diffusion Model](https://mlsanity.com/paper/2402.16143/) (2024)
- [AutoTransition: Learning to Recommend Video Transition Effects](https://mlsanity.com/paper/2207.13479/) (2022)
- [Match Cutting: Finding Cuts with Smooth Visual Transitions](https://mlsanity.com/paper/2210.05766/) (2022)
- [Temporal and Contextual Transformer for Multi-Camera Editing of TV Shows](https://mlsanity.com/paper/2210.08737/) (2022)
- [Long-range Multimodal Pretraining for Movie Understanding](https://mlsanity.com/paper/2308.09775/) (2023)
- [CinePreGen: Camera Controllable Video Previsualization via Engine-powered Diffusion](https://mlsanity.com/paper/2408.17424/) (2024)
- [VEU-Bench: Towards Comprehensive Understanding of Video Editing](https://mlsanity.com/paper/2504.17828/) (2025)
- [From Long Videos to Engaging Clips: A Human-Inspired Video Editing Framework with Multimodal Narrative Understanding](https://mlsanity.com/paper/2507.02790/) (2025)
- [A Benchmark and Multi-Agent System for Instruction-driven Cinematic Video Compilation](https://mlsanity.com/paper/2604.10456/) (2026)
- [Exploring Temporal Granularity in Self-Supervised Video Representation Learning](https://mlsanity.com/paper/2112.04480/) (2021)
- [Generative Timelines for Instructed Visual Assembly](https://mlsanity.com/paper/2411.12293/) (2024)
- [MatchDiffusion: Training-free Generation of Match-cuts](https://mlsanity.com/paper/2411.18677/) (2024)
- [Audio Match Cutting: Finding and Creating Matching Audio Transitions in Movies and Videos](https://mlsanity.com/paper/2408.10998/) (2024)
- [ShotFinder: Imagination-Driven Open-Domain Video Shot Retrieval via Web Search](https://mlsanity.com/paper/2601.23232/) (2026)

## Abstract (arXiv)

Video content creation keeps growing at an incredible pace; yet, creating engaging stories remains challenging and requires non-trivial video editing expertise. Many video editing components are astonishingly hard to automate primarily due to the lack of raw video materials. This paper focuses on a new task for computational video editing, namely the task of raking cut plausibility. Our key idea is to leverage content that has already been edited to learn fine-grained audiovisual patterns that trigger cuts. To do this, we first collected a data source of more than 10K videos, from which we extract more than 255K cuts. We devise a model that learns to discriminate between real and artificial cuts via contrastive learning. We set up a new task and a set of baselines to benchmark video cut generation. We observe that our proposed model outperforms the baselines by large margins. To demonstrate our model in real-world applications, we conduct human studies in a collection of unedited videos. The results show that our model does a better job at cutting than random and alternative baselines.