# Point and Ask: Incorporating Pointing into Visual Question Answering

arXiv: https://arxiv.org/abs/2011.13681 · HTML: https://mlsanity.com/paper/2011.13681/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Nov 2020.

## Built on by (most cited first, 30 of 39)

- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts](https://mlsanity.com/paper/2312.00784/) (2023)
- [MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices](https://mlsanity.com/paper/2312.16886/) (2023)
- [SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understanding](https://mlsanity.com/paper/2406.10100/) (2024)
- [Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want](https://mlsanity.com/paper/2403.20271/) (2024)
- [From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models](https://mlsanity.com/paper/2310.08825/) (2023)
- [MoVA: Adapting Mixture of Vision Experts to Multimodal Context](https://mlsanity.com/paper/2404.13046/) (2024)
- [NExT-Chat: An LMM for Chat, Detection and Segmentation](https://mlsanity.com/paper/2311.04498/) (2023)
- [GROUNDHOG: Grounding Large Language Models to Holistic Segmentation](https://mlsanity.com/paper/2402.16846/) (2024)
- [ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning](https://mlsanity.com/paper/2307.09474/) (2023)
- [Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model](https://mlsanity.com/paper/2312.12423/) (2023)
- [Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs](https://mlsanity.com/paper/2310.00582/) (2023)
- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [Visual Instruction Tuning with Polite Flamingo](https://mlsanity.com/paper/2307.01003/) (2023)
- [Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach](https://mlsanity.com/paper/2412.18108/) (2024)
- [List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs](https://mlsanity.com/paper/2404.16375/) (2024)
- [Visual Instruction Tuning towards General-Purpose Multimodal Model: A Survey](https://mlsanity.com/paper/2312.16602/) (2023)
- [Voila-A: Aligning Vision-Language Models with User's Gaze Attention](https://mlsanity.com/paper/2401.09454/) (2023)
- [TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal Large Language Models](https://mlsanity.com/paper/2404.09204/) (2024)
- [LOVA3: Learning to Visual Question Answering, Asking and Assessment](https://mlsanity.com/paper/2405.14974/) (2024)
- [Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgery](https://mlsanity.com/paper/2408.04958/) (2024)
- [From Training-Free to Adaptive: Empirical Insights into MLLMs' Understanding of Detection Information](https://mlsanity.com/paper/2401.17981/) (2024)
- [ToolVQA: A Dataset for Multi-step Reasoning VQA with External Tools](https://mlsanity.com/paper/2508.03284/) (2025)
- [Genixer: Empowering Multimodal Large Language Models as a Powerful Data Generator](https://mlsanity.com/paper/2312.06731/) (2023)
- [SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Models](https://mlsanity.com/paper/2403.13263/) (2024)
- [Towards Understanding Visual Grounding in Visual Language Models](https://mlsanity.com/paper/2509.10345/) (2025)
- [AssistSR: Task-oriented Video Segment Retrieval for Personal AI Assistant](https://mlsanity.com/paper/2111.15050/) (2021)
- [A Gaze-grounded Visual Question Answering Dataset for Clarifying Ambiguous Japanese Questions](https://mlsanity.com/paper/2403.17545/) (2024)
- [HAMMR: HierArchical MultiModal React agents for generic VQA](https://mlsanity.com/paper/2404.05465/) (2024)

## Abstract (arXiv)

Visual Question Answering (VQA) has become one of the key benchmarks of visual recognition progress. Multiple VQA extensions have been explored to better simulate real-world settings: different question formulations, changing training and test distributions, conversational consistency in dialogues, and explanation-based answering. In this work, we further expand this space by considering visual questions that include a spatial point of reference. Pointing is a nearly universal gesture among humans, and real-world VQA is likely to involve a gesture towards the target region. Concretely, we (1) introduce and motivate point-input questions as an extension of VQA, (2) define three novel classes of questions within this space, and (3) for each class, introduce both a benchmark dataset and a series of baseline models to handle its unique challenges. There are two key distinctions from prior work. First, we explicitly design the benchmarks to require the point input, i.e., we ensure that the visual question cannot be answered accurately without the spatial reference. Second, we explicitly explore the more realistic point spatial input rather than the standard but unnatural bounding box input. Through our exploration we uncover and address several visual recognition challenges, including the ability to infer human intent, reason both locally and globally about the image, and effectively combine visual, language and spatial inputs. Code is available at: https://github.com/princetonvisualai/pointingqa .