# Look, Read and Ask: Learning to Ask Questions by Reading Text in Images

arXiv: https://arxiv.org/abs/2211.12950 · HTML: https://mlsanity.com/paper/2211.12950/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 43 references that are in the map.
- First posted on arXiv 23 Nov 2022.

## Builds on (most cited first, 30 of 30)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [TextCaps: a Dataset for Image Captioning with Reading Comprehension](https://mlsanity.com/paper/2003.12462/) (2020)
- [Scene Text Visual Question Answering](https://mlsanity.com/paper/1905.13648/) (2019)
- [Character Region Awareness for Text Detection](https://mlsanity.com/paper/1904.01941/) (2019)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [Reading Text in the Wild with Convolutional Neural Networks](https://mlsanity.com/paper/1412.1842/) (2014)
- [Scene Text Detection and Recognition: The Deep Learning Era](https://mlsanity.com/paper/1811.04256/) (2018)
- [Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA](https://mlsanity.com/paper/1911.06258/) (2019)
- [Rosetta: Large scale system for text detection and recognition in images](https://mlsanity.com/paper/1910.05085/) (2019)
- [Discriminability objective for training descriptive captions](https://mlsanity.com/paper/1803.04376/) (2018)
- [Generating Natural Questions About an Image](https://mlsanity.com/paper/1603.06059/) (2016)
- [Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text](https://mlsanity.com/paper/2003.13962/) (2020)
- [Visual Question Generation as Dual Task of Visual Question Answering](https://mlsanity.com/paper/1709.07192/) (2017)
- [Information Maximizing Visual Question Generation](https://mlsanity.com/paper/1903.11207/) (2019)
- [Generating Factoid Questions With Recurrent Neural Networks: The 30M Factoid Question-Answer Corpus](https://mlsanity.com/paper/1603.06807/) (2016)
- [Learning by Asking Questions](https://mlsanity.com/paper/1712.01238/) (2017)
- [Creativity: Generating Diverse Questions using Variational Autoencoders](https://mlsanity.com/paper/1704.03493/) (2017)
- [Automatic Generation of Grounded Visual Questions](https://mlsanity.com/paper/1612.06530/) (2016)
- [Visual Curiosity: Learning to Ask Questions to Learn Visual Recognition](https://mlsanity.com/paper/1810.00912/) (2018)
- [Deep Bayesian Network for Visual Question Generation](https://mlsanity.com/paper/2001.08779/) (2020)
- [StacMR: Scene-Text Aware Cross-Modal Retrieval](https://mlsanity.com/paper/2012.04329/) (2020)
- [Plan, Attend, Generate: Planning for Sequence-to-Sequence Models](https://mlsanity.com/paper/1711.10462/) (2017)

## Built on by (most cited first, 5 of 5)

- [Towards Cross-modal Retrieval in Chinese Cultural Heritage Documents: Dataset and Solution](https://mlsanity.com/paper/2505.10921/) (2025)
- [Understanding Video Scenes through Text: Insights from Text-based Video Question Answering](https://mlsanity.com/paper/2309.01380/) (2023)
- [Making the V in Text-VQA Matter](https://mlsanity.com/paper/2308.00295/) (2023)
- [Weakly Supervised Visual Question Answer Generation](https://mlsanity.com/paper/2306.06622/) (2023)
- [When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs](https://mlsanity.com/paper/2509.16633/) (2025)

## Abstract (arXiv)

We present a novel problem of text-based visual question generation or TextVQG in short. Given the recent growing interest of the document image analysis community in combining text understanding with conversational artificial intelligence, e.g., text-based visual question answering, TextVQG becomes an important task. TextVQG aims to generate a natural language question for a given input image and an automatically extracted text also known as OCR token from it such that the OCR token is an answer to the generated question. TextVQG is an essential ability for a conversational agent. However, it is challenging as it requires an in-depth understanding of the scene and the ability to semantically bridge the visual content with the text present in the image. To address TextVQG, we present an OCR consistent visual question generation model that Looks into the visual content, Reads the scene text, and Asks a relevant and meaningful natural language question. We refer to our proposed model as OLRA. We perform an extensive evaluation of OLRA on two public benchmarks and compare them against baselines. Our model OLRA automatically generates questions similar to the public text-based visual question answering datasets that were curated manually. Moreover, we significantly outperform baseline approaches on the performance measures popularly used in text generation literature.