# Context-VQA: Towards Context-Aware and Purposeful Visual Question Answering

arXiv: https://arxiv.org/abs/2307.15745 · HTML: https://mlsanity.com/paper/2307.15745/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 18 references that are in the map.
- First posted on arXiv 28 Jul 2023.

## Builds on (most cited first, 11 of 11)

- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [Visual7W: Grounded Question Answering in Images](https://mlsanity.com/paper/1511.03416/) (2015)
- [Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics](https://mlsanity.com/paper/2205.10646/) (2022)
- [Image Captioning as an Assistive Technology: Lessons Learned from VizWiz 2020 Challenge](https://mlsanity.com/paper/2012.11696/) (2020)
- [Concadia: Towards Image-Based Text Generation with a Purpose](https://mlsanity.com/paper/2104.08376/) (2021)
- [What's Different between Visual Question Answering for Machine "Understanding" Versus for Accessibility?](https://mlsanity.com/paper/2210.14966/) (2022)

## Built on by (most cited first, 6 of 6)

- [Long-Form Answers to Visual Questions from Blind and Low Vision People](https://mlsanity.com/paper/2408.06303/) (2024)
- [Fully Authentic Visual Question Answering Dataset from Online Communities](https://mlsanity.com/paper/2311.15562/) (2023)
- [Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answering](https://mlsanity.com/paper/2502.04469/) (2025)
- [RJUA-MedDQA: A Multimodal Benchmark for Medical Document Question Answering and Clinical Reasoning](https://mlsanity.com/paper/2402.14840/) (2024)
- [DriveSOTIF: Advancing Perception SOTIF Through Multimodal Large Language Models](https://mlsanity.com/paper/2505.07084/) (2025)
- [WISER: Wider Search, Deeper Thinking, and Adaptive Fusion for Training-Free Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2602.23029/) (2026)

## Abstract (arXiv)

Visual question answering (VQA) has the potential to make the Internet more accessible in an interactive way, allowing people who cannot see images to ask questions about them. However, multiple studies have shown that people who are blind or have low-vision prefer image explanations that incorporate the context in which an image appears, yet current VQA datasets focus on images in isolation. We argue that VQA models will not fully succeed at meeting people's needs unless they take context into account. To further motivate and analyze the distinction between different contexts, we introduce Context-VQA, a VQA dataset that pairs images with contexts, specifically types of websites (e.g., a shopping website). We find that the types of questions vary systematically across contexts. For example, images presented in a travel context garner 2 times more "Where?" questions, and images on social media and news garner 2.8 and 1.8 times more "Who?" questions than the average. We also find that context effects are especially important when participants can't see the image. These results demonstrate that context affects the types of questions asked and that VQA models should be context-sensitive to better meet people's needs, especially in accessibility settings.