# KnowIT VQA: Answering Knowledge-Based Questions about Videos

arXiv: https://arxiv.org/abs/1910.10706 · HTML: https://mlsanity.com/paper/1910.10706/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2019.

## Built on by (most cited first, 30 of 53)

- [A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge](https://mlsanity.com/paper/2206.01718/) (2022)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [Zero-Shot Video Question Answering via Frozen Bidirectional Language Models](https://mlsanity.com/paper/2206.08155/) (2022)
- [MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding](https://mlsanity.com/paper/2406.14515/) (2024)
- [Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videos](https://mlsanity.com/paper/2110.05122/) (2021)
- [SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Events](https://mlsanity.com/paper/2103.15538/) (2021)
- [Video Question Answering: Datasets, Algorithms and Challenges](https://mlsanity.com/paper/2203.01225/) (2022)
- [REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering](https://mlsanity.com/paper/2206.01201/) (2022)
- [3D Question Answering](https://mlsanity.com/paper/2112.08359/) (2021)
- [Sports-QA: A Large-Scale Video Question Answering Benchmark for Complex and Professional Sports](https://mlsanity.com/paper/2401.01505/) (2024)
- [Gender and Racial Bias in Visual Question Answering Datasets](https://mlsanity.com/paper/2205.08148/) (2022)
- [The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning](https://mlsanity.com/paper/2202.04800/) (2022)
- [FunQA: Towards Surprising Video Comprehension](https://mlsanity.com/paper/2306.14899/) (2023)
- [Distribution-Aware Prompt Tuning for Vision-Language Models](https://mlsanity.com/paper/2309.03406/) (2023)
- [Learning to Answer Visual Questions from Web Videos](https://mlsanity.com/paper/2205.05019/) (2022)
- [From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities](https://mlsanity.com/paper/2311.00308/) (2023)
- [i-Code: An Integrative and Composable Multimodal Learning Framework](https://mlsanity.com/paper/2205.01818/) (2022)
- [Modern Question Answering Datasets and Benchmarks: A Survey](https://mlsanity.com/paper/2206.15030/) (2022)
- [Tem-adapter: Adapting Image-Text Pretraining for Video Question Answer](https://mlsanity.com/paper/2308.08414/) (2023)
- [ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmark](https://mlsanity.com/paper/2501.05031/) (2025)
- [Learning Musical Representations for Music Performance Question Answering](https://mlsanity.com/paper/2502.06710/) (2025)
- [Video Question Answering with Iterative Video-Text Co-Tokenization](https://mlsanity.com/paper/2208.00934/) (2022)
- [WorldQA: Multimodal World Knowledge in Videos through Long-Chain Reasoning](https://mlsanity.com/paper/2405.03272/) (2024)
- [Referring Atomic Video Action Recognition](https://mlsanity.com/paper/2407.01872/) (2024)
- [Knowledge Graph-extended Retrieval Augmented Generation for Question Answering](https://mlsanity.com/paper/2504.08893/) (2025)
- [Recent Advances in Video Question Answering: A Review of Datasets and Methods](https://mlsanity.com/paper/2101.05954/) (2021)
- [A survey on VQA_Datasets and Approaches](https://mlsanity.com/paper/2105.00421/) (2021)
- [Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models](https://mlsanity.com/paper/2503.18923/) (2025)
- [WildQA: In-the-Wild Video Question Answering](https://mlsanity.com/paper/2209.06650/) (2022)
- [TOGA: Temporally Grounded Open-Ended Video QA with Weak Supervision](https://mlsanity.com/paper/2506.09445/) (2025)

## Abstract (arXiv)

We propose a novel video understanding task by fusing knowledge-based and video question answering. First, we introduce KnowIT VQA, a video dataset with 24,282 human-generated question-answer pairs about a popular sitcom. The dataset combines visual, textual and temporal coherence reasoning together with knowledge-based questions, which need of the experience obtained from the viewing of the series to be answered. Second, we propose a video understanding model by combining the visual and textual video content with specific knowledge about the show. Our main findings are: (i) the incorporation of knowledge produces outstanding improvements for VQA in video, and (ii) the performance on KnowIT VQA still lags well behind human accuracy, indicating its usefulness for studying current video modelling limitations.