# Inverse Visual Question Answering: A New Benchmark and VQA Diagnosis Tool

arXiv: https://arxiv.org/abs/1803.06936 · HTML: https://mlsanity.com/paper/1803.06936/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Mar 2018.

## Built on by (most cited first, 9 of 9)

- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [Deep Learning for Omnidirectional Vision: A Survey and New Perspectives](https://mlsanity.com/paper/2205.10468/) (2022)
- [Entity-enhanced Adaptive Reconstruction Network for Weakly Supervised Referring Expression Grounding](https://mlsanity.com/paper/2207.08386/) (2022)
- [Visual Question Answering: which investigated applications?](https://mlsanity.com/paper/2103.02937/) (2021)
- [ConVQG: Contrastive Visual Question Generation with Multimodal Guidance](https://mlsanity.com/paper/2402.12846/) (2024)
- [Evaluation Under Imperfect Benchmarks and Ratings: A Case Study in Text Simplification](https://mlsanity.com/paper/2504.09394/) (2025)
- [Ask Questions with Double Hints: Visual Question Generation with Answer-awareness and Region-reference](https://mlsanity.com/paper/2407.05100/) (2024)
- [SimpsonsVQA: Enhancing Inquiry-Based Learning with a Tailored Dataset](https://mlsanity.com/paper/2410.22648/) (2024)
- [Reinforcement Explanation Learning](https://mlsanity.com/paper/2111.13406/) (2021)

## Abstract (arXiv)

In recent years, visual question answering (VQA) has become topical. The premise of VQA's significance as a benchmark in AI, is that both the image and textual question need to be well understood and mutually grounded in order to infer the correct answer. However, current VQA models perhaps `understand' less than initially hoped, and instead master the easier task of exploiting cues given away in the question and biases in the answer distribution. In this paper we propose the inverse problem of VQA (iVQA). The iVQA task is to generate a question that corresponds to a given image and answer pair. We propose a variational iVQA model that can generate diverse, grammatically correct and content correlated questions that match the given answer. Based on this model, we show that iVQA is an interesting benchmark for visuo-linguistic understanding, and a more challenging alternative to VQA because an iVQA model needs to understand the image better to be successful. As a second contribution, we show how to use iVQA in a novel reinforcement learning framework to diagnose any existing VQA model by way of exposing its belief set: the set of question-answer pairs that the VQA model would predict true for a given image. This provides a completely new window into what VQA models `believe' about images. We show that existing VQA models have more erroneous beliefs than previously thought, revealing their intrinsic weaknesses. Suggestions are then made on how to address these weaknesses going forward.