# Uncovering Temporal Context for Video Question and Answering

arXiv: https://arxiv.org/abs/1511.04670 · HTML: https://mlsanity.com/paper/1511.04670/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2015.

## Built on by (most cited first, 3 of 3)

- [Improving Visual Question Answering Models through Robustness Analysis and In-Context Learning with a Chain of Basic Questions](https://mlsanity.com/paper/2304.03147/) (2023)
- [TimeLogic: A Temporal Logic Benchmark for Video QA](https://mlsanity.com/paper/2501.07214/) (2025)
- [Scene Detection Policies and Keyframe Extraction Strategies for Large-Scale Video Analysis](https://mlsanity.com/paper/2506.00667/) (2025)

## Abstract (arXiv)

In this work, we introduce Video Question Answering in temporal domain to infer the past, describe the present and predict the future. We present an encoder-decoder approach using Recurrent Neural Networks to learn temporal structures of videos and introduce a dual-channel ranking loss to answer multiple-choice questions. We explore approaches for finer understanding of video content using question form of "fill-in-the-blank", and managed to collect 109,895 video clips with duration over 1,000 hours from TACoS, MPII-MD, MEDTest 14 datasets, while the corresponding 390,744 questions are generated from annotations. Extensive experiments demonstrate that our approach significantly outperforms the compared baselines.