# MovieNet: A Holistic Dataset for Movie Understanding

arXiv: https://arxiv.org/abs/2007.10937 · HTML: https://mlsanity.com/paper/2007.10937/

## Key facts

- Cited by 206 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jul 2020.

## Built on by (most cited first, 30 of 206)

- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models](https://mlsanity.com/paper/2311.17043/) (2023)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [MLVU: Benchmarking Multi-task Long Video Understanding](https://mlsanity.com/paper/2406.04264/) (2024)
- [Video Understanding with Large Language Models: A Survey](https://mlsanity.com/paper/2312.17432/) (2023)
- [Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams](https://mlsanity.com/paper/2406.08085/) (2024)
- [Streaming Long Video Understanding with Large Language Models](https://mlsanity.com/paper/2405.16009/) (2024)
- [Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution](https://mlsanity.com/paper/2409.12961/) (2024)
- [Towards Long-Form Video Understanding](https://mlsanity.com/paper/2106.11310/) (2021)
- [StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding](https://mlsanity.com/paper/2411.03628/) (2024)
- [OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?](https://mlsanity.com/paper/2501.05510/) (2025)
- [RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness](https://mlsanity.com/paper/2405.17220/) (2024)
- [Streaming Video Question-Answering with In-context Video KV-Cache Retrieval](https://mlsanity.com/paper/2503.00540/) (2025)
- [HourVideo: 1-Hour Video-Language Understanding](https://mlsanity.com/paper/2411.04998/) (2024)
- [Long Movie Clip Classification with State-Space Video Models](https://mlsanity.com/paper/2204.01692/) (2022)
- [Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performance](https://mlsanity.com/paper/2410.16261/) (2024)
- [MileBench: Benchmarking MLLMs in Long Context](https://mlsanity.com/paper/2404.18532/) (2024)
- [Flash-VStream: Efficient Real-Time Understanding for Long Video Streams](https://mlsanity.com/paper/2506.23825/) (2025)
- [MovieChat+: Question-aware Sparse Memory for Long Video Question Answering](https://mlsanity.com/paper/2404.17176/) (2024)
- [Category-Level 6D Object Pose Estimation via Cascaded Relation and Recurrent Reconstruction Networks](https://mlsanity.com/paper/2108.08755/) (2021)
- [DrVideo: Document Retrieval Based Long Video Understanding](https://mlsanity.com/paper/2406.12846/) (2024)
- [Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models](https://mlsanity.com/paper/2504.15271/) (2025)
- [Towards Understanding Camera Motions in Any Video](https://mlsanity.com/paper/2504.15376/) (2025)
- [Visual Semantic Role Labeling for Video Understanding](https://mlsanity.com/paper/2104.00990/) (2021)
- [Needle In A Multimodal Haystack](https://mlsanity.com/paper/2406.07230/) (2024)
- [InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understanding](https://mlsanity.com/paper/2506.15745/) (2025)
- [SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding](https://mlsanity.com/paper/2502.10810/) (2025)
- [Efficient Movie Scene Detection using State-Space Transformers](https://mlsanity.com/paper/2212.14427/) (2022)

## Abstract (arXiv)

Recent years have seen remarkable advances in visual understanding. However, how to understand a story-based long video with artistic styles, e.g. movie, remains challenging. In this paper, we introduce MovieNet -- a holistic dataset for movie understanding. MovieNet contains 1,100 movies with a large amount of multi-modal data, e.g. trailers, photos, plot descriptions, etc. Besides, different aspects of manual annotations are provided in MovieNet, including 1.1M characters with bounding boxes and identities, 42K scene boundaries, 2.5K aligned description sentences, 65K tags of place and action, and 92K tags of cinematic style. To the best of our knowledge, MovieNet is the largest dataset with richest annotations for comprehensive movie understanding. Based on MovieNet, we set up several benchmarks for movie understanding from different angles. Extensive experiments are executed on these benchmarks to show the immeasurable value of MovieNet and the gap of current approaches towards comprehensive movie understanding. We believe that such a holistic dataset would promote the researches on story-based long video understanding and beyond. MovieNet will be published in compliance with regulations at https://movienet.github.io.