# Watching the News: Towards VideoQA Models that can Read

arXiv: https://arxiv.org/abs/2211.05588 · HTML: https://mlsanity.com/paper/2211.05588/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 31 of its 41 references that are in the map.
- First posted on arXiv 10 Nov 2022.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering](https://mlsanity.com/paper/1906.02467/) (2019)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training](https://mlsanity.com/paper/2005.00200/) (2020)
- [DVQA: Understanding Data Visualizations via Question Answering](https://mlsanity.com/paper/1801.08163/) (2018)
- [Scene Text Visual Question Answering](https://mlsanity.com/paper/1905.13648/) (2019)
- [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017)
- [MovieQA: Understanding Stories in Movies through Question-Answering](https://mlsanity.com/paper/1512.02902/) (2015)
- [Just Ask: Learning to Answer Questions from Millions of Narrated Videos](https://mlsanity.com/paper/2012.00451/) (2020)
- [A Joint Sequence Fusion Model for Video Question Answering and Retrieval](https://mlsanity.com/paper/1808.02559/) (2018)
- [Revealing Single Frame Bias for Video-and-Language Learning](https://mlsanity.com/paper/2206.03428/) (2022)
- [Visual Question Answering: A Survey of Methods and Datasets](https://mlsanity.com/paper/1607.05910/) (2016)
- [Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA](https://mlsanity.com/paper/1911.06258/) (2019)
- [TAP: Text-Aware Pre-training for Text-VQA and Text-Caption](https://mlsanity.com/paper/2012.04638/) (2020)
- [On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answering](https://mlsanity.com/paper/2002.10215/) (2020)
- [LaTr: Layout-Aware Transformer for Scene-Text VQA](https://mlsanity.com/paper/2112.12494/) (2021)
- [Document Collection Visual Question Answering](https://mlsanity.com/paper/2104.14336/) (2021)
- [A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering](https://mlsanity.com/paper/1611.07810/) (2016)

## Built on by (most cited first, 20 of 20)

- [mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding](https://mlsanity.com/paper/2409.03420/) (2024)
- [Reading Between the Lanes: Text VideoQA on the Road](https://mlsanity.com/paper/2307.03948/) (2023)
- [MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios](https://mlsanity.com/paper/2505.21333/) (2025)
- [Video Question Answering for People with Visual Impairments Using an Egocentric 360-Degree Camera](https://mlsanity.com/paper/2405.19794/) (2024)
- [A Survey of Video Datasets for Grounded Event Understanding](https://mlsanity.com/paper/2406.09646/) (2024)
- [Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion](https://mlsanity.com/paper/2306.08889/) (2023)
- [MultiVENT: Multilingual Videos of Events with Aligned Natural Text](https://mlsanity.com/paper/2307.03153/) (2023)
- [Do Current Video LLMs Have Strong OCR Abilities? A Preliminary Study](https://mlsanity.com/paper/2412.20613/) (2024)
- [An LMM for Efficient Video Understanding via Reinforced Compression of Video Cubes](https://mlsanity.com/paper/2504.15270/) (2025)
- [Video-R4: Reinforcing Text-Rich Video Reasoning with Visual Rumination](https://mlsanity.com/paper/2511.17490/) (2025)
- [Scene-Text Grounding for Text-Based Video Question Answering](https://mlsanity.com/paper/2409.14319/) (2024)
- [Understanding Video Scenes through Text: Insights from Text-based Video Question Answering](https://mlsanity.com/paper/2309.01380/) (2023)
- [DocSLM: A Small Vision-Language Model for Long Multimodal Document Understanding](https://mlsanity.com/paper/2511.11313/) (2025)
- [Making the V in Text-VQA Matter](https://mlsanity.com/paper/2308.00295/) (2023)
- [ViTexQA: A Multi-Frame Temporal Perception Dataset for Video Text Question Answering](https://mlsanity.com/paper/2606.24602/) (2026)
- [Weakly Supervised Visual Question Answer Generation](https://mlsanity.com/paper/2306.06622/) (2023)
- [QTG-VQA: Question-Type-Guided Architectural for VideoQA Systems](https://mlsanity.com/paper/2409.09348/) (2024)
- [TextVidBench: A Benchmark for Long Video Scene Text Understanding](https://mlsanity.com/paper/2506.04983/) (2025)
- [VNU-Bench: A Benchmarking Dataset for Multi-Source Multimodal News Video Understanding](https://mlsanity.com/paper/2601.03434/) (2026)
- [ClearText-Video: A Large-Scale Text-Centric Video Dataset Bridging Video Restoration and Scene-Text Enhancement](https://mlsanity.com/paper/2608.28784/) (2026)

## Abstract (arXiv)

Video Question Answering methods focus on commonsense reasoning and visual cognition of objects or persons and their interactions over time. Current VideoQA approaches ignore the textual information present in the video. Instead, we argue that textual information is complementary to the action and provides essential contextualisation cues to the reasoning process. To this end, we propose a novel VideoQA task that requires reading and understanding the text in the video. To explore this direction, we focus on news videos and require QA systems to comprehend and answer questions about the topics presented by combining visual and textual cues in the video. We introduce the ``NewsVideoQA'' dataset that comprises more than 8,600 QA pairs on 3,000+ news videos obtained from diverse news channels from around the world. We demonstrate the limitations of current Scene Text VQA and VideoQA methods and propose ways to incorporate scene text information into VideoQA methods.