# The Promise of Premise: Harnessing Question Premises in Visual Question Answering

arXiv: https://arxiv.org/abs/1705.00601 · HTML: https://mlsanity.com/paper/1705.00601/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 May 2017.

## Built on by (most cited first, 11 of 11)

- [Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly](https://mlsanity.com/paper/2204.13631/) (2022)
- [See, Say, and Segment: Teaching LMMs to Overcome False Premises](https://mlsanity.com/paper/2312.08366/) (2023)
- [Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness](https://mlsanity.com/paper/2407.01942/) (2024)
- [Unanswerable Questions about Images and Texts](https://mlsanity.com/paper/2102.06793/) (2021)
- [Do What? Teaching Vision-Language-Action Models to Reject the Impossible](https://mlsanity.com/paper/2508.16292/) (2025)
- [Toward Unsupervised Realistic Visual Question Answering](https://mlsanity.com/paper/2303.05068/) (2023)
- [YTCommentQA: Video Question Answerability in Instructional Videos](https://mlsanity.com/paper/2401.17343/) (2024)
- [Evaluating and Improving Interactions with Hazy Oracles](https://mlsanity.com/paper/2110.10206/) (2021)
- [Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless Predictions](https://mlsanity.com/paper/2405.11145/) (2024)
- [SimpsonsVQA: Enhancing Inquiry-Based Learning with a Tailored Dataset](https://mlsanity.com/paper/2410.22648/) (2024)
- [Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models](https://mlsanity.com/paper/2609.04720/) (2026)

## Abstract (arXiv)

In this paper, we make a simple observation that questions about images often contain premises - objects and relationships implied by the question - and that reasoning about premises can help Visual Question Answering (VQA) models respond more intelligently to irrelevant or previously unseen questions. When presented with a question that is irrelevant to an image, state-of-the-art VQA models will still answer purely based on learned language biases, resulting in non-sensical or even misleading answers. We note that a visual question is irrelevant to an image if at least one of its premises is false (i.e. not depicted in the image). We leverage this observation to construct a dataset for Question Relevance Prediction and Explanation (QRPE) by searching for false premises. We train novel question relevance detection models and show that models that reason about premises consistently outperform models that do not. We also find that forcing standard VQA models to reason about premises during training can lead to improvements on tasks requiring compositional reasoning.