# UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Models

arXiv: https://arxiv.org/abs/2310.10942 · HTML: https://mlsanity.com/paper/2310.10942/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 84 references that are in the map.
- First posted on arXiv 17 Oct 2023.

## Builds on (most cited first, 30 of 65)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)

## Built on by (most cited first, 25 of 25)

- [EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarios](https://mlsanity.com/paper/2412.04447/) (2024)
- [The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defense](https://mlsanity.com/paper/2411.08410/) (2024)
- [Describe-then-Reason: Improving Multimodal Mathematical Reasoning through Visual Comprehension Training](https://mlsanity.com/paper/2404.14604/) (2024)
- [Unsolvable Problem Detection: Robust Understanding Evaluation for Large Multimodal Models](https://mlsanity.com/paper/2403.20331/) (2024)
- [MoHoBench: Assessing Honesty of Multimodal Large Language Models via Unanswerable Visual Questions](https://mlsanity.com/paper/2507.21503/) (2025)
- [Enhancing Descriptive Captions with Visual Attributes for Multimodal Perception](https://mlsanity.com/paper/2412.14233/) (2024)
- [LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Models](https://mlsanity.com/paper/2410.01620/) (2024)
- [CLIP-UP: CLIP-Based Unanswerable Problem Detection for Visual Question Answering](https://mlsanity.com/paper/2501.01371/) (2025)
- [AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answering](https://mlsanity.com/paper/2601.12248/) (2026)
- [TRAPSBench: Vision-Language Models Encode but Fail to Express Epistemic Restraint](https://mlsanity.com/paper/2608.13167/) (2026)
- [CLASH: A Benchmark for Cross-Modal Contradiction Detection](https://mlsanity.com/paper/2511.19199/) (2025)
- [When Not to Answer: Evaluating Prompts on GPT Models for Effective Abstention in Unanswerable Math Word Problems](https://mlsanity.com/paper/2410.13029/) (2024)
- [LMOD+: A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in Ophthalmology](https://mlsanity.com/paper/2509.25620/) (2025)
- [ProactiveBench: Benchmarking Proactiveness in Multimodal Large Language Models](https://mlsanity.com/paper/2603.19466/) (2026)
- [Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?](https://mlsanity.com/paper/2605.30557/) (2026)
- [Mitigating Low-Level Visual Hallucinations Requires Self-Awareness: Database, Model and Training Strategy](https://mlsanity.com/paper/2503.20673/) (2025)
- [Multimodal AI for Gastrointestinal Diagnostics: Tackling VQA in MEDVQA-GI 2025](https://mlsanity.com/paper/2507.14544/) (2025)
- [Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoning](https://mlsanity.com/paper/2602.14518/) (2026)
- [AQuA: Toward Strategic Response Generation for Ambiguous Visual Questions](https://mlsanity.com/paper/2603.07394/) (2026)
- [VB: Visibility Benchmark for Visibility and Perspective Reasoning in Images](https://mlsanity.com/paper/2603.06680/) (2026)
- [Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models](https://mlsanity.com/paper/2609.04720/) (2026)
- [From Knowing to Abstaining: Bridging the Representation-Action Gap in Vision-Language Models](https://mlsanity.com/paper/2609.32653/) (2026)
- [When Does an Image Determine the Answer? Benchmarking Visual Answerability across Charts and Scenes](https://mlsanity.com/paper/2609.34480/) (2026)
- [Lost in Contrast: Instruction-Preserving Decoding for Reliable Visual Question Answering](https://mlsanity.com/paper/2610.16722/) (2026)
- [Visual Dependence Does Not Imply Answerability: A Systematic Analysis of Answerability Signals in Frozen InstructBLIP](https://mlsanity.com/paper/2610.17902/) (2026)

## Abstract (arXiv)

Teaching Visual Question Answering (VQA) models to refrain from answering unanswerable questions is necessary for building a trustworthy AI system. Existing studies, though have explored various aspects of VQA but somewhat ignored this particular attribute. This paper aims to bridge the research gap by contributing a comprehensive dataset, called UNK-VQA. The dataset is specifically designed to address the challenge of questions that models do not know. To this end, we first augment the existing data via deliberate perturbations on either the image or question. In specific, we carefully ensure that the question-image semantics remain close to the original unperturbed distribution. By this means, the identification of unanswerable questions becomes challenging, setting our dataset apart from others that involve mere image replacement. We then extensively evaluate the zero- and few-shot performance of several emerging multi-modal large models and discover their significant limitations when applied to our dataset. Additionally, we also propose a straightforward method to tackle these unanswerable questions. This dataset, we believe, will serve as a valuable benchmark for enhancing the abstention capability of VQA models, thereby leading to increased trustworthiness of AI systems. We have made the dataset (https://github.com/guoyang9/UNK-VQA) available to facilitate further exploration in this area.