# MovieQA: Understanding Stories in Movies through Question-Answering

arXiv: https://arxiv.org/abs/1512.02902 · HTML: https://mlsanity.com/paper/1512.02902/

## Key facts

- Cited by 304 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Dec 2015.

## Built on by (most cited first, 30 of 304)

- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge](https://mlsanity.com/paper/2206.01718/) (2022)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [LVBench: An Extreme Long Video Understanding Benchmark](https://mlsanity.com/paper/2406.08035/) (2024)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation](https://mlsanity.com/paper/1811.10092/) (2018)
- [Zero-Shot Video Question Answering via Frozen Bidirectional Language Models](https://mlsanity.com/paper/2206.08155/) (2022)
- [VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding](https://mlsanity.com/paper/2403.11481/) (2024)
- [STAR: A Benchmark for Situated Reasoning in Real-World Videos](https://mlsanity.com/paper/2405.09711/) (2024)
- [Learning to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2203.14072/) (2022)
- [MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding](https://mlsanity.com/paper/2406.14515/) (2024)
- [SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images](https://mlsanity.com/paper/2301.04883/) (2023)
- [A Simple LLM Framework for Long-Range Video Question-Answering](https://mlsanity.com/paper/2312.17235/) (2023)
- [Towards Long-Form Video Understanding](https://mlsanity.com/paper/2106.11310/) (2021)
- [Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning](https://mlsanity.com/paper/2402.11435/) (2024)
- [AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning](https://mlsanity.com/paper/2103.16002/) (2021)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [EgoTaskQA: Understanding Human Tasks in Egocentric Videos](https://mlsanity.com/paper/2210.03929/) (2022)
- [CinePile: A Long Video Question Answering Dataset and Benchmark](https://mlsanity.com/paper/2405.08813/) (2024)
- [WebQA: Multihop and Multimodal QA](https://mlsanity.com/paper/2109.00590/) (2021)
- [Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models](https://mlsanity.com/paper/2311.16103/) (2023)
- [Long Movie Clip Classification with State-Space Video Models](https://mlsanity.com/paper/2204.01692/) (2022)
- [TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment](https://mlsanity.com/paper/2108.09980/) (2021)
- [WebSRC: A Dataset for Web-Based Structural Reading Comprehension](https://mlsanity.com/paper/2101.09465/) (2021)
- [QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension](https://mlsanity.com/paper/2107.12708/) (2021)
- [Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videos](https://mlsanity.com/paper/2110.05122/) (2021)
- [SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Events](https://mlsanity.com/paper/2103.15538/) (2021)
- [Video Question Answering: Datasets, Algorithms and Challenges](https://mlsanity.com/paper/2203.01225/) (2022)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)

## Abstract (arXiv)

We introduce the MovieQA dataset which aims to evaluate automatic story comprehension from both video and text. The dataset consists of 14,944 questions about 408 movies with high semantic diversity. The questions range from simpler "Who" did "What" to "Whom", to "Why" and "How" certain events occurred. Each question comes with a set of five possible answers; a correct one and four deceiving answers provided by human annotators. Our dataset is unique in that it contains multiple sources of information -- video clips, plots, subtitles, scripts, and DVS. We analyze our data through various statistics and methods. We further extend existing QA techniques to show that question-answering with such open-ended semantics is hard. We make this data set public along with an evaluation benchmark to encourage inspiring work in this challenging domain.