# Understanding Video Scenes through Text: Insights from Text-based Video Question Answering

arXiv: https://arxiv.org/abs/2309.01380 · HTML: https://mlsanity.com/paper/2309.01380/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 4 Sep 2023.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering](https://mlsanity.com/paper/1906.02467/) (2019)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [InfographicVQA](https://mlsanity.com/paper/2104.12756/) (2021)
- [TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering](https://mlsanity.com/paper/1704.04497/) (2017)
- [Scene Text Visual Question Answering](https://mlsanity.com/paper/1905.13648/) (2019)
- [MovieQA: Understanding Stories in Movies through Question-Answering](https://mlsanity.com/paper/1512.02902/) (2015)
- [Revealing Single Frame Bias for Video-and-Language Learning](https://mlsanity.com/paper/2206.03428/) (2022)
- [Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA](https://mlsanity.com/paper/1911.06258/) (2019)
- [Reading Between the Lanes: Text VideoQA on the Road](https://mlsanity.com/paper/2307.03948/) (2023)
- [Watching the News: Towards VideoQA Models that can Read](https://mlsanity.com/paper/2211.05588/) (2022)
- [Look, Read and Ask: Learning to Ask Questions by Reading Text in Images](https://mlsanity.com/paper/2211.12950/) (2022)
- [Making the V in Text-VQA Matter](https://mlsanity.com/paper/2308.00295/) (2023)

## Built on by (most cited first, 2 of 2)

- [EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering](https://mlsanity.com/paper/2502.07411/) (2025)
- [TSalV360: A Method and Dataset for Text-driven Saliency Detection in 360-Degrees Videos](https://mlsanity.com/paper/2509.26208/) (2025)

## Abstract (arXiv)

Researchers have extensively studied the field of vision and language, discovering that both visual and textual content is crucial for understanding scenes effectively. Particularly, comprehending text in videos holds great significance, requiring both scene text understanding and temporal reasoning. This paper focuses on exploring two recently introduced datasets, NewsVideoQA and M4-ViteVQA, which aim to address video question answering based on textual content. The NewsVideoQA dataset contains question-answer pairs related to the text in news videos, while M4-ViteVQA comprises question-answer pairs from diverse categories like vlogging, traveling, and shopping. We provide an analysis of the formulation of these datasets on various levels, exploring the degree of visual understanding and multi-frame comprehension required for answering the questions. Additionally, the study includes experimentation with BERT-QA, a text-only model, which demonstrates comparable performance to the original methods on both datasets, indicating the shortcomings in the formulation of these datasets. Furthermore, we also look into the domain adaptation aspect by examining the effectiveness of training on M4-ViteVQA and evaluating on NewsVideoQA and vice-versa, thereby shedding light on the challenges and potential benefits of out-of-domain training.