# Scene-Text Grounding for Text-Based Video Question Answering

arXiv: https://arxiv.org/abs/2409.14319 · HTML: https://mlsanity.com/paper/2409.14319/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 64 references that are in the map.
- First posted on arXiv 22 Sep 2024.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [Scene Text Visual Question Answering](https://mlsanity.com/paper/1905.13648/) (2019)
- [Counterfactual VQA: A Cause-Effect Look at Language Bias](https://mlsanity.com/paper/2006.04315/) (2020)
- [Self-Chained Image-Language Model for Video Localization and Question Answering](https://mlsanity.com/paper/2305.06988/) (2023)
- [Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition](https://mlsanity.com/paper/2103.06495/) (2021)
- [TVQA+: Spatio-Temporal Grounding for Video Question Answering](https://mlsanity.com/paper/1904.11574/) (2019)
- [Can I Trust Your Answer? Visually Grounded Video Question Answering](https://mlsanity.com/paper/2309.01327/) (2023)
- [Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA](https://mlsanity.com/paper/1911.06258/) (2019)
- [Invariant Grounding for Video Question Answering](https://mlsanity.com/paper/2206.02349/) (2022)
- [MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answering](https://mlsanity.com/paper/2212.09522/) (2022)
- [VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation](https://mlsanity.com/paper/1708.04686/) (2017)
- [Benchmarking Micro-action Recognition: Dataset, Methods, and Applications](https://mlsanity.com/paper/2403.05234/) (2024)
- [Grounding Answers for Visual Questions Asked by Visually Impaired People](https://mlsanity.com/paper/2202.01993/) (2022)
- [MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2203.09138/) (2022)
- [DualVGR: A Dual-Visual Graph Reasoning Unit for Video Question Answering](https://mlsanity.com/paper/2107.04768/) (2021)
- [Reducing Language Biases in Visual Question Answering with Visually-Grounded Question Encoder](https://mlsanity.com/paper/2007.06198/) (2020)
- [Contrastive Video Question Answering via Video Graph Transformer](https://mlsanity.com/paper/2302.13668/) (2023)
- [A Bilingual, OpenWorld Video Text Dataset and End-to-end Video Text Spotter with Transformer](https://mlsanity.com/paper/2112.04888/) (2021)
- [Reading Between the Lanes: Text VideoQA on the Road](https://mlsanity.com/paper/2307.03948/) (2023)

## Built on by (most cited first, 5 of 5)

- [EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering](https://mlsanity.com/paper/2502.07411/) (2025)
- [Enhancing Partially Relevant Video Retrieval with Robust Alignment Learning](https://mlsanity.com/paper/2509.01383/) (2025)
- [SafeGuard: A Multi-Agent Perception-Reasoning Framework for Social-Risk AI-Generated Video Detection](https://mlsanity.com/paper/2607.03069/) (2026)
- [GroundSight: Augmenting Vision-Language Models with Grounding Information and De-hallucination](https://mlsanity.com/paper/2509.25669/) (2025)
- [ClearText-Video: A Large-Scale Text-Centric Video Dataset Bridging Video Restoration and Scene-Text Enhancement](https://mlsanity.com/paper/2608.28784/) (2026)

## Abstract (arXiv)

Existing efforts in text-based video question answering (TextVideoQA) are criticized for their opaque decisionmaking and heavy reliance on scene-text recognition. In this paper, we propose to study Grounded TextVideoQA by forcing models to answer questions and spatio-temporally localize the relevant scene-text regions, thus decoupling QA from scenetext recognition and promoting research towards interpretable QA. The task has three-fold significance. First, it encourages scene-text evidence versus other short-cuts for answer predictions. Second, it directly accepts scene-text regions as visual answers, thus circumventing the problem of ineffective answer evaluation by stringent string matching. Third, it isolates the challenges inherited in VideoQA and scene-text recognition. This enables the diagnosis of the root causes for failure predictions, e.g., wrong QA or wrong scene-text recognition? To achieve Grounded TextVideoQA, we propose the T2S-QA model that highlights a disentangled temporal-to-spatial contrastive learning strategy for weakly-supervised scene-text grounding and grounded TextVideoQA. To facilitate evaluation, we construct a new dataset ViTXT-GQA which features 52K scene-text bounding boxes within 2.2K temporal segments related to 2K questions and 729 videos. With ViTXT-GQA, we perform extensive experiments and demonstrate the severe limitations of existing techniques in Grounded TextVideoQA. While T2S-QA achieves superior results, the large performance gap with human leaves ample space for improvement. Our further analysis of oracle scene-text inputs posits that the major challenge is scene-text recognition. To advance the research of Grounded TextVideoQA, our dataset and code are at https://github.com/zhousheng97/ViTXT-GQA.git