# TVQA+: Spatio-Temporal Grounding for Video Question Answering

arXiv: https://arxiv.org/abs/1904.11574 · HTML: https://mlsanity.com/paper/1904.11574/

## Key facts

- Cited by 154 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2019.

## Built on by (most cited first, 30 of 154)

- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [Retrieval-Augmented Generation for AI-Generated Content: A Survey](https://mlsanity.com/paper/2402.19473/) (2024)
- [MERLOT: Multimodal Neural Script Knowledge Models](https://mlsanity.com/paper/2106.02636/) (2021)
- [Visual Spatial Reasoning](https://mlsanity.com/paper/2205.00363/) (2022)
- [Zero-Shot Video Question Answering via Frozen Bidirectional Language Models](https://mlsanity.com/paper/2206.08155/) (2022)
- [STAR: A Benchmark for Situated Reasoning in Real-World Videos](https://mlsanity.com/paper/2405.09711/) (2024)
- [VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling](https://mlsanity.com/paper/2111.12681/) (2021)
- [SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images](https://mlsanity.com/paper/2301.04883/) (2023)
- [All in One: Exploring Unified Video-Language Pre-training](https://mlsanity.com/paper/2203.07303/) (2022)
- [Can I Trust Your Answer? Visually Grounded Video Question Answering](https://mlsanity.com/paper/2309.01327/) (2023)
- [AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning](https://mlsanity.com/paper/2103.16002/) (2021)
- [Vision-Language Pre-training: Basics, Recent Advances, and Future Trends](https://mlsanity.com/paper/2210.09263/) (2022)
- [EgoTaskQA: Understanding Human Tasks in Egocentric Videos](https://mlsanity.com/paper/2210.03929/) (2022)
- [VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation](https://mlsanity.com/paper/2106.04632/) (2021)
- [Retrieving Multimodal Information for Augmented Generation: A Survey](https://mlsanity.com/paper/2303.10868/) (2023)
- [SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Events](https://mlsanity.com/paper/2103.15538/) (2021)
- [Video Question Answering: Datasets, Algorithms and Challenges](https://mlsanity.com/paper/2203.01225/) (2022)
- [An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling](https://mlsanity.com/paper/2209.01540/) (2022)
- [QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries](https://mlsanity.com/paper/2107.09609/) (2021)
- [Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QA](https://mlsanity.com/paper/2406.09396/) (2024)
- [V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoning](https://mlsanity.com/paper/2503.11495/) (2025)
- [HierVL: Learning Hierarchical Video-Language Embeddings](https://mlsanity.com/paper/2301.02311/) (2023)
- [ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding System](https://mlsanity.com/paper/2304.14407/) (2023)
- [ComPhy: Compositional Physical Reasoning of Objects and Events from Videos](https://mlsanity.com/paper/2205.01089/) (2022)
- [DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)](https://mlsanity.com/paper/2401.08392/) (2024)
- [Natural Language Video Localization: A Revisit in Span-based Question Answering Framework](https://mlsanity.com/paper/2102.13558/) (2021)
- [MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answering](https://mlsanity.com/paper/2212.09522/) (2022)
- [Grounding 'Grounding' in NLP](https://mlsanity.com/paper/2106.02192/) (2021)

## Abstract (arXiv)

We present the task of Spatio-Temporal Video Question Answering, which requires intelligent systems to simultaneously retrieve relevant moments and detect referenced visual concepts (people and objects) to answer natural language questions about videos. We first augment the TVQA dataset with 310.8K bounding boxes, linking depicted objects to visual concepts in questions and answers. We name this augmented version as TVQA+. We then propose Spatio-Temporal Answerer with Grounded Evidence (STAGE), a unified framework that grounds evidence in both spatial and temporal domains to answer questions about videos. Comprehensive experiments and analyses demonstrate the effectiveness of our framework and how the rich annotations in our TVQA+ dataset can contribute to the question answering task. Moreover, by performing this joint task, our model is able to produce insightful and interpretable spatio-temporal attention visualizations. Dataset and code are publicly available at: http: //tvqa.cs.unc.edu, https://github.com/jayleicn/TVQAplus