# VQA Therapy: Exploring Answer Differences by Visually Grounding Answers

arXiv: https://arxiv.org/abs/2308.11662 · HTML: https://mlsanity.com/paper/2308.11662/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 35 references that are in the map.
- First posted on arXiv 21 Aug 2023.

## Builds on (most cited first, 29 of 29)

- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [Visual7W: Grounded Question Answering in Images](https://mlsanity.com/paper/1511.03416/) (2015)
- [Modeling Context Between Objects for Referring Expression Understanding](https://mlsanity.com/paper/1608.00525/) (2016)
- [Universal Instance Perception as Object Discovery and Retrieval](https://mlsanity.com/paper/2303.06674/) (2023)
- [Multimodal Explanations: Justifying Decisions and Pointing to the Evidence](https://mlsanity.com/paper/1802.08129/) (2018)
- [SeqTR: A Simple yet Universal Network for Visual Grounding](https://mlsanity.com/paper/2203.16265/) (2022)
- [PhraseCut: Language-based Image Segmentation in the Wild](https://mlsanity.com/paper/2008.01187/) (2020)
- [Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?](https://mlsanity.com/paper/1606.03556/) (2016)
- [VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation](https://mlsanity.com/paper/1708.04686/) (2017)
- [Grounding Answers for Visual Questions Asked by Visually Impaired People](https://mlsanity.com/paper/2202.01993/) (2022)
- [Why Does a Visual Question Have Different Answers?](https://mlsanity.com/paper/1908.04342/) (2019)
- [Interpretable Visual Question Answering by Visual Grounding from Attention Supervision Mining](https://mlsanity.com/paper/1808.00265/) (2018)
- [AiR: Attention with Reasoning Capability](https://mlsanity.com/paper/2007.14419/) (2020)
- [Found a Reason for me? Weakly-supervised Grounded Visual Question Answering using Capsules](https://mlsanity.com/paper/2105.04836/) (2021)
- [Vision Skills Needed to Answer Visual Questions](https://mlsanity.com/paper/2010.03160/) (2020)
- [A First Look: Towards Explainable TextVQA Models via Visual and Textual Explanations](https://mlsanity.com/paper/2105.02626/) (2021)
- [Why Did the Chicken Cross the Road? Rephrasing and Analyzing Ambiguous Questions in VQA](https://mlsanity.com/paper/2211.07516/) (2022)
- [REX: Reasoning-aware and Grounded Explanation](https://mlsanity.com/paper/2203.06107/) (2022)

## Built on by (most cited first, 14 of 14)

- [Long-Form Answers to Visual Questions from Blind and Low Vision People](https://mlsanity.com/paper/2408.06303/) (2024)
- [Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Users](https://mlsanity.com/paper/2503.22610/) (2025)
- [RefChartQA: Grounding Visual Answer on Chart Images through Instruction Tuning](https://mlsanity.com/paper/2503.23131/) (2025)
- [Towards Understanding Visual Grounding in Visual Language Models](https://mlsanity.com/paper/2509.10345/) (2025)
- [Acknowledging Focus Ambiguity in Visual Questions](https://mlsanity.com/paper/2501.02201/) (2025)
- [Selectively Answering Visual Questions](https://mlsanity.com/paper/2406.00980/) (2024)
- [Scene-Text Grounding for Text-Based Video Question Answering](https://mlsanity.com/paper/2409.14319/) (2024)
- [Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory](https://mlsanity.com/paper/2603.02663/) (2026)
- [BLaVe-CoT: Consistency-Aware Visual Question Answering for Blind and Low Vision Users](https://mlsanity.com/paper/2509.06010/) (2025)
- [StackOverflowVQA: Stack Overflow Visual Question Answering Dataset](https://mlsanity.com/paper/2405.10736/) (2024)
- [Second Place Solution of WSDM2023 Toloka Visual Question Answering Challenge](https://mlsanity.com/paper/2407.04255/) (2024)
- [Hierarchical Instance Tracking to Balance Privacy Preservation with Accessible Information](https://mlsanity.com/paper/2512.10102/) (2025)
- [Explicit Logic Channel for Validation and Enhancement of MLLMs on Zero-Shot Tasks](https://mlsanity.com/paper/2603.11689/) (2026)
- [Multimodal Plant Root Phenotyping with Integration of 3D Skeleton Extraction and Language Analysis](https://mlsanity.com/paper/2608.03109/) (2026)

## Abstract (arXiv)

Visual question answering is a task of predicting the answer to a question about an image. Given that different people can provide different answers to a visual question, we aim to better understand why with answer groundings. We introduce the first dataset that visually grounds each unique answer to each visual question, which we call VQAAnswerTherapy. We then propose two novel problems of predicting whether a visual question has a single answer grounding and localizing all answer groundings. We benchmark modern algorithms for these novel problems to show where they succeed and struggle. The dataset and evaluation server can be found publicly at https://vizwiz.org/tasks-and-datasets/vqa-answer-therapy/.