# Question Relevance in VQA: Identifying Non-Visual And False-Premise Questions

arXiv: https://arxiv.org/abs/1606.06622 · HTML: https://mlsanity.com/paper/1606.06622/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2016.

## Built on by (most cited first, 16 of 16)

- [Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly](https://mlsanity.com/paper/2204.13631/) (2022)
- [A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility](https://mlsanity.com/paper/2202.02312/) (2022)
- [COLA: A Benchmark for Compositional Text-to-image Retrieval](https://mlsanity.com/paper/2305.03689/) (2023)
- [See, Say, and Segment: Teaching LMMs to Overcome False Premises](https://mlsanity.com/paper/2312.08366/) (2023)
- [On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations](https://mlsanity.com/paper/2204.12708/) (2022)
- [Unmasking Deceptive Visuals: Benchmarking Multimodal Large Language Models on Misleading Chart Question Answering](https://mlsanity.com/paper/2503.18172/) (2025)
- [Unanswerable Questions about Images and Texts](https://mlsanity.com/paper/2102.06793/) (2021)
- [Do What? Teaching Vision-Language-Action Models to Reject the Impossible](https://mlsanity.com/paper/2508.16292/) (2025)
- [Improving Users' Mental Model with Attention-directed Counterfactual Edits](https://mlsanity.com/paper/2110.06863/) (2021)
- [What You Say Is What You Show: Visual Narration Detection in Instructional Videos](https://mlsanity.com/paper/2301.02307/) (2023)
- [Toward Unsupervised Realistic Visual Question Answering](https://mlsanity.com/paper/2303.05068/) (2023)
- [YTCommentQA: Video Question Answerability in Instructional Videos](https://mlsanity.com/paper/2401.17343/) (2024)
- [Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless Predictions](https://mlsanity.com/paper/2405.11145/) (2024)
- [SimpsonsVQA: Enhancing Inquiry-Based Learning with a Tailored Dataset](https://mlsanity.com/paper/2410.22648/) (2024)
- [Context-Aware Multi-Turn Visual-Textual Reasoning in LVLMs via Dynamic Memory and Adaptive Visual Guidance](https://mlsanity.com/paper/2509.05669/) (2025)
- [Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models](https://mlsanity.com/paper/2609.04720/) (2026)

## Abstract (arXiv)

Visual Question Answering (VQA) is the task of answering natural-language questions about images. We introduce the novel problem of determining the relevance of questions to images in VQA. Current VQA models do not reason about whether a question is even related to the given image (e.g. What is the capital of Argentina?) or if it requires information from external resources to answer correctly. This can break the continuity of a dialogue in human-machine interaction. Our approaches for determining relevance are composed of two stages. Given an image and a question, (1) we first determine whether the question is visual or not, (2) if visual, we determine whether the question is relevant to the given image or not. Our approaches, based on LSTM-RNNs, VQA model uncertainty, and caption-question similarity, are able to outperform strong baselines on both relevance tasks. We also present human studies showing that VQA models augmented with such question relevance reasoning are perceived as more intelligent, reasonable, and human-like.