# Knowledge-Based Video Question Answering with Unsupervised Scene Descriptions

arXiv: https://arxiv.org/abs/2007.08751 · HTML: https://mlsanity.com/paper/2007.08751/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jul 2020.

## Built on by (most cited first, 15 of 15)

- [Learning to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2203.14072/) (2022)
- [SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Events](https://mlsanity.com/paper/2103.15538/) (2021)
- [Video Question Answering: Datasets, Algorithms and Challenges](https://mlsanity.com/paper/2203.01225/) (2022)
- [Temporal Query Networks for Fine-grained Video Understanding](https://mlsanity.com/paper/2104.09496/) (2021)
- [REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering](https://mlsanity.com/paper/2206.01201/) (2022)
- [Exploiting Long-Term Dependencies for Generating Dynamic Scene Graphs](https://mlsanity.com/paper/2112.09828/) (2021)
- [i-Code: An Integrative and Composable Multimodal Learning Framework](https://mlsanity.com/paper/2205.01818/) (2022)
- [Learning to Cut by Watching Movies](https://mlsanity.com/paper/2108.04294/) (2021)
- [Video-based Human-Object Interaction Detection from Tubelet Tokens](https://mlsanity.com/paper/2206.01908/) (2022)
- [A survey on VQA_Datasets and Approaches](https://mlsanity.com/paper/2105.00421/) (2021)
- [On the hidden treasure of dialog in video question answering](https://mlsanity.com/paper/2103.14517/) (2021)
- [Transferring Domain-Agnostic Knowledge in Video Question Answering](https://mlsanity.com/paper/2110.13395/) (2021)
- [LiVLR: A Lightweight Visual-Linguistic Reasoning Framework for Video Question Answering](https://mlsanity.com/paper/2111.14547/) (2021)
- [A Picture May Be Worth a Hundred Words for Visual Question Answering](https://mlsanity.com/paper/2106.13445/) (2021)
- [Uncertainty-based Visual Question Answering: Estimating Semantic Inconsistency between Image and Knowledge Base](https://mlsanity.com/paper/2207.13242/) (2022)

## Abstract (arXiv)

To understand movies, humans constantly reason over the dialogues and actions shown in specific scenes and relate them to the overall storyline already seen. Inspired by this behaviour, we design ROLL, a model for knowledge-based video story question answering that leverages three crucial aspects of movie understanding: dialog comprehension, scene reasoning, and storyline recalling. In ROLL, each of these tasks is in charge of extracting rich and diverse information by 1) processing scene dialogues, 2) generating unsupervised video scene descriptions, and 3) obtaining external knowledge in a weakly supervised fashion. To answer a given question correctly, the information generated by each inspired-cognitive task is encoded via Transformers and fused through a modality weighting mechanism, which balances the information from the different sources. Exhaustive evaluation demonstrates the effectiveness of our approach, which yields a new state-of-the-art on two challenging video question answering datasets: KnowIT VQA and TVQA+.