# Information Maximizing Visual Question Generation

arXiv: https://arxiv.org/abs/1903.11207 · HTML: https://mlsanity.com/paper/1903.11207/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Mar 2019.

## Built on by (most cited first, 30 of 40)

- [TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering](https://mlsanity.com/paper/2303.11897/) (2023)
- [All You May Need for VQA are Image Captions](https://mlsanity.com/paper/2205.01883/) (2022)
- [Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Task](https://mlsanity.com/paper/2208.12037/) (2022)
- [Generative Language Models for Paragraph-Level Question Generation](https://mlsanity.com/paper/2210.03992/) (2022)
- [LOVA3: Learning to Visual Question Answering, Asking and Assessment](https://mlsanity.com/paper/2405.14974/) (2024)
- [Improving Biomedical Information Retrieval with Neural Retrievers](https://mlsanity.com/paper/2201.07745/) (2022)
- [Q: How to Specialize Large Vision-Language Models to Data-Scarce VQA Tasks? A: Self-Train on Unlabeled Images!](https://mlsanity.com/paper/2306.03932/) (2023)
- [Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answering](https://mlsanity.com/paper/2109.06122/) (2021)
- [Guiding Visual Question Generation](https://mlsanity.com/paper/2110.08226/) (2021)
- [Why Did the Chicken Cross the Road? Rephrasing and Analyzing Ambiguous Questions in VQA](https://mlsanity.com/paper/2211.07516/) (2022)
- [Simple Baselines for Interactive Video Retrieval with Questions and Answers](https://mlsanity.com/paper/2308.10402/) (2023)
- [K-VQG: Knowledge-aware Visual Question Generation for Common-sense Acquisition](https://mlsanity.com/paper/2203.07890/) (2022)
- [Cross-Modal Generative Augmentation for Visual Question Answering](https://mlsanity.com/paper/2105.04780/) (2021)
- [The Dialog Must Go On: Improving Visual Dialog via Generative Self-Training](https://mlsanity.com/paper/2205.12502/) (2022)
- [End-to-End Video Question-Answer Generation with Generator-Pretester Network](https://mlsanity.com/paper/2101.01447/) (2021)
- [Asking More Informative Questions for Grounded Retrieval](https://mlsanity.com/paper/2311.08584/) (2023)
- [One VLM to Keep it Learning: Generation and Balancing for Data-free Continual Visual Question Answering](https://mlsanity.com/paper/2411.02210/) (2024)
- [Look, Read and Ask: Learning to Ask Questions by Reading Text in Images](https://mlsanity.com/paper/2211.12950/) (2022)
- [A Survey on Neural Question Generation: Methods, Applications, and Prospects](https://mlsanity.com/paper/2402.18267/) (2024)
- [Landmark-Guided Cross-Speaker Lip Reading with Mutual Information Regularization](https://mlsanity.com/paper/2403.16071/) (2024)
- [ConVQG: Contrastive Visual Question Generation with Multimodal Guidance](https://mlsanity.com/paper/2402.12846/) (2024)
- [Sketch-based Creativity Support Tools using Deep Learning](https://mlsanity.com/paper/2111.09991/) (2021)
- [Latent Variable Models for Visual Question Answering](https://mlsanity.com/paper/2101.06399/) (2021)
- [Cross-modal Image Retrieval with Deep Mutual Information Maximization](https://mlsanity.com/paper/2103.06032/) (2021)
- [Learning by Asking Questions for Knowledge-based Novel Object Recognition](https://mlsanity.com/paper/2210.05879/) (2022)
- [Diagram-Driven Course Questions Generation](https://mlsanity.com/paper/2411.17771/) (2024)
- [Questions beyond Pixels: Integrating Commonsense Knowledge in Visual Question Generation for Remote Sensing](https://mlsanity.com/paper/2602.19217/) (2026)
- [Goal-driven text descriptions for images](https://mlsanity.com/paper/2108.12575/) (2021)
- [Visual Question Generation in Bengali](https://mlsanity.com/paper/2310.08187/) (2023)
- [Ask Questions with Double Hints: Visual Question Generation with Answer-awareness and Region-reference](https://mlsanity.com/paper/2407.05100/) (2024)

## Abstract (arXiv)

Though image-to-sequence generation models have become overwhelmingly popular in human-computer communications, they suffer from strongly favoring safe generic questions ("What is in this picture?"). Generating uninformative but relevant questions is not sufficient or useful. We argue that a good question is one that has a tightly focused purpose --- one that is aimed at expecting a specific type of response. We build a model that maximizes mutual information between the image, the expected answer and the generated question. To overcome the non-differentiability of discrete natural language tokens, we introduce a variational continuous latent space onto which the expected answers project. We regularize this latent space with a second latent space that ensures clustering of similar answers. Even when we don't know the expected answer, this second latent space can generate goal-driven questions specifically aimed at extracting objects ("what is the person throwing"), attributes, ("What kind of shirt is the person wearing?"), color ("what color is the frisbee?"), material ("What material is the frisbee?"), etc. We quantitatively show that our model is able to retain information about an expected answer category, resulting in more diverse, goal-driven questions. We launch our model on a set of real world images and extract previously unseen visual concepts.