# Making the V in Text-VQA Matter

arXiv: https://arxiv.org/abs/2308.00295 · HTML: https://mlsanity.com/paper/2308.00295/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 1 Aug 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering](https://mlsanity.com/paper/1712.00377/) (2017)
- [Scene Text Visual Question Answering](https://mlsanity.com/paper/1905.13648/) (2019)
- [Counterfactual VQA: A Cause-Effect Look at Language Bias](https://mlsanity.com/paper/2006.04315/) (2020)
- [RUBi: Reducing Unimodal Biases in Visual Question Answering](https://mlsanity.com/paper/1906.10169/) (2019)
- [Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded](https://mlsanity.com/paper/1902.03751/) (2019)
- [Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA](https://mlsanity.com/paper/1911.06258/) (2019)
- [TAP: Text-Aware Pre-training for Text-VQA and Text-Caption](https://mlsanity.com/paper/2012.04638/) (2020)
- [Rosetta: Large scale system for text detection and recognition in images](https://mlsanity.com/paper/1910.05085/) (2019)
- [Overcoming Language Priors with Self-supervised Learning for Visual Question Answering](https://mlsanity.com/paper/2012.11528/) (2020)
- [Unshuffling Data for Improved Generalization](https://mlsanity.com/paper/2002.11894/) (2020)
- [Reducing Language Biases in Visual Question Answering with Visually-Grounded Question Encoder](https://mlsanity.com/paper/2007.06198/) (2020)
- [Structured Multimodal Attentions for TextVQA](https://mlsanity.com/paper/2006.00753/) (2020)
- [Watching the News: Towards VideoQA Models that can Read](https://mlsanity.com/paper/2211.05588/) (2022)
- [A First Look: Towards Explainable TextVQA Models via Visual and Textual Explanations](https://mlsanity.com/paper/2105.02626/) (2021)
- [Localize, Group, and Select: Boosting Text-VQA by Scene Text Modeling](https://mlsanity.com/paper/2108.08965/) (2021)
- [Look, Read and Ask: Learning to Ask Questions by Reading Text in Images](https://mlsanity.com/paper/2211.12950/) (2022)

## Built on by (most cited first, 1 of 1)

- [Understanding Video Scenes through Text: Insights from Text-based Video Question Answering](https://mlsanity.com/paper/2309.01380/) (2023)

## Abstract (arXiv)

Text-based VQA aims at answering questions by reading the text present in the images. It requires a large amount of scene-text relationship understanding compared to the VQA task. Recent studies have shown that the question-answer pairs in the dataset are more focused on the text present in the image but less importance is given to visual features and some questions do not require understanding the image. The models trained on this dataset predict biased answers due to the lack of understanding of visual context. For example, in questions like "What is written on the signboard?", the answer predicted by the model is always "STOP" which makes the model to ignore the image. To address these issues, we propose a method to learn visual features (making V matter in TextVQA) along with the OCR features and question features using VQA dataset as external knowledge for Text-based VQA. Specifically, we combine the TextVQA dataset and VQA dataset and train the model on this combined dataset. Such a simple, yet effective approach increases the understanding and correlation between the image features and text present in the image, which helps in the better answering of questions. We further test the model on different datasets and compare their qualitative and quantitative results.