mlsanityData to 7 Oct 2026

A Gaze-grounded Visual Question Answering Dataset for Clarifying Ambiguous Japanese Questions

arXiv 2403.17545 ↗First posted 26 Mar 2024cs.CL cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Situated conversations, which refer to visual information as visual question answering (VQA), often contain ambiguities caused by reliance on directive information. This problem is exacerbated because some languages, such as Japanese, often omit subjective or objective terms. Such ambiguities in questions are often clarified by the contexts in conversational situations, such as joint attention with a user or user gaze information. In this study, we propose the Gaze-grounded VQA dataset (GazeVQA) that clarifies ambiguous questions using gaze information by focusing on a clarification process complemented by gaze information. We also propose a method that utilizes gaze target estimation results to improve the accuracy of GazeVQA tasks. Our experimental results showed that the proposed method improved the performance in some cases of a VQA system on GazeVQA and identified some typical problems of GazeVQA tasks that need to be improved.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 23 of its 41 references that are in the map. First posted on arXiv 26 Mar 2024.

Builds on

23 of 41 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  10. 2021Image Segmentation Using Text and Image Promptscited by 509
  11. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  12. 2016Visual Dialogcited by 377
11 more
  1. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  2. 2016Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?cited by 111
  3. 2020Detecting Attended Visual Targets in Videocited by 76
  4. 2018Connecting Gaze, Scene, and Attention: Generalized Attention Estimation via Joint Modeling of Gaze and Scene Saliencycited by 53
  5. 2017STAIR Captions: Constructing a Large-Scale Japanese Image Caption Datasetcited by 47
  6. 2020Point and Ask: Incorporating Pointing into Visual Question Answeringcited by 39
  7. 2020History for Visual Dialog: Do we really need it?cited by 32
  8. 2021GOO: A Dataset for Gaze Object Prediction in Retail Environmentscited by 26
  9. 2019What You See is What You Get: Visual Pronoun Coreference Resolution in Dialoguescited by 18
  10. 2021VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answeringcited by 16
  11. 2023DeePoint: Visual Pointing Recognition and Direction Estimationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videoscited by 8
  2. 2025Integrating Cognitive Processing Signals into Language Models: A Review of Advances, Applications and Future Directionscited by 2
  3. 2025Take That for Me: Multimodal Exophora Resolution with Interactive Questioning for Ambiguous Out-of-View Instructionscited by 1
  4. 2026IRIS: Intent Resolution via Inference-time Saccades for Open-Ended VQA in Large Vision-Language Modelscited by 1
  5. 2025J-ORA: A Framework and Multimodal Dataset for Japanese Object Identification, Reference, Action Prediction in Robot Perceptioncited by 0
  6. 2026LaViSA: A Language and Vision Structural Ambiguity Benchmarkcited by 0
  7. 2026WeaveAgent: A Two-Stage Tool-Routing Agent for Ultra-High-Resolution Remote Sensing Imagerycited by 0