# VisionTrap: Unanswerable Questions On Visual Data

arXiv: https://arxiv.org/abs/2507.17262 · HTML: https://mlsanity.com/paper/2507.17262/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 23 Jul 2025.

## Builds on (most cited first, 27 of 27)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)
- [Visual Question Answering: A Survey of Methods and Datasets](https://mlsanity.com/paper/1607.05910/) (2016)
- [An Analysis of Visual Question Answering Algorithms](https://mlsanity.com/paper/1703.09684/) (2017)
- [Attention Sorting Combats Recency Bias In Long Context Language Models](https://mlsanity.com/paper/2310.01427/) (2023)
- [Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly](https://mlsanity.com/paper/2204.13631/) (2022)
- [Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models](https://mlsanity.com/paper/2407.16221/) (2024)
- [GANDiffFace: Controllable Generation of Synthetic Datasets for Face Recognition with Realistic Variations](https://mlsanity.com/paper/2305.19962/) (2023)
- [Camera-to-Robot Pose Estimation from a Single Image](https://mlsanity.com/paper/1911.09231/) (2019)
- [Benchmarking Hallucination in Large Language Models based on Unanswerable Math Word Problem](https://mlsanity.com/paper/2403.03558/) (2024)
- [FFHQ-UV: Normalized Facial UV-Texture Dataset for 3D Face Reconstruction](https://mlsanity.com/paper/2211.13874/) (2022)
- [Characterizing LLM Abstention Behavior in Science QA with Context Perturbations](https://mlsanity.com/paper/2404.12452/) (2024)
- [Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization](https://mlsanity.com/paper/2205.12191/) (2022)
- [MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering](https://mlsanity.com/paper/2503.18491/) (2025)
- [Survey of Visual Question Answering: Datasets and Techniques](https://mlsanity.com/paper/1705.03865/) (2017)
- [CLIP-UP: CLIP-Based Unanswerable Problem Detection for Visual Question Answering](https://mlsanity.com/paper/2501.01371/) (2025)
- [DARE: Diverse Visual Question Answering with Robustness Evaluation](https://mlsanity.com/paper/2409.18023/) (2024)
- [Interpretable Visual Question Answering via Reasoning Supervision](https://mlsanity.com/paper/2309.03726/) (2023)

## Built on by (most cited first, 1 of 1)

- [TRAPSBench: Vision-Language Models Encode but Fail to Express Epistemic Restraint](https://mlsanity.com/paper/2608.13167/) (2026)

## Abstract (arXiv)

Visual Question Answering (VQA) has been a widely studied topic, with extensive research focusing on how VLMs respond to answerable questions based on real-world images. However, there has been limited exploration of how these models handle unanswerable questions, particularly in cases where they should abstain from providing a response. This research investigates VQA performance on unrealistically generated images or asking unanswerable questions, assessing whether models recognize the limitations of their knowledge or attempt to generate incorrect answers. We introduced a dataset, VisionTrap, comprising three categories of unanswerable questions across diverse image types: (1) hybrid entities that fuse objects and animals, (2) objects depicted in unconventional or impossible scenarios, and (3) fictional or non-existent figures. The questions posed are logically structured yet inherently unanswerable, testing whether models can correctly recognize their limitations. Our findings highlight the importance of incorporating such questions into VQA benchmarks to evaluate whether models tend to answer, even when they should abstain.