mlsanityData to 7 Oct 2026

Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?

arXiv 1606.03556 ↗First posted 11 Jun 2016cs.CV cs.CLcited by 111

Not on the map. Why?

Abstract (arXiv)

We conduct large-scale studies on `human attention' in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images. We design and test multiple game-inspired novel attention-annotation interfaces that require the subject to sharpen regions of a blurred image to answer a question. Thus, we introduce the VQA-HAT (Human ATtention) dataset. We evaluate attention maps generated by state-of-the-art VQA models against human attention both qualitatively (via visualizations) and quantitatively (via rank-order correlation). Overall, our experiments show that current attention models in VQA do not seem to be looking at the same regions as humans.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Jun 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 31; 2022: 26; 2023: 18; 2024: 17; 2025: 8; 2026: 11.

111 papers.

Sort:

By area:

  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  2. 2021Medical Visual Question Answering: A Surveycited by 93
  3. 2021Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answeringcited by 71
  4. 2022A General Survey on Attention Mechanisms in Deep Learningcited by 64
  5. 2022Grounding Answers for Visual Questions Asked by Visually Impaired Peoplecited by 47
  6. 2022Deep Learning Approaches on Image Captioning: A Reviewcited by 43
  7. 2022Going Beyond XAI: A Systematic Survey for Explanation-Guided Learningcited by 35
  8. 2021Greedy Gradient Ensemble for Robust Visual Question Answeringcited by 34
  9. 2021Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answeringcited by 33
  10. 2023Voila-A: Aligning Vision-Language Models with User's Gaze Attentioncited by 28
  11. 2022Aligning Eyes between Humans and Deep Neural Network through Interactive Attention Alignmentcited by 27
  12. 2023Towards Trustable Skin Cancer Diagnosis via Rewriting Model's Decisioncited by 27
38 more
  1. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  2. 2023Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directionscited by 25
  3. 2021Separating Skills and Concepts for Novel Visual Question Answeringcited by 24
  4. 2023Robust Visual Question Answering: Datasets, Methods, and Future Challengescited by 23
  5. 2021Taxonomy of Machine Learning Safety: A Survey and Primercited by 22
  6. 2022RES: A Robust Framework for Guiding Visual Explanationcited by 22
  7. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
  8. 2021Multimodal Integration of Human-Like Attention in Visual Question Answeringcited by 20
  9. 2021Human Attention in Fine-grained Classificationcited by 20
  10. 2022Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze?cited by 20
  11. 2023Visual Question Answering: A Survey on Techniques and Common Trends in Recent Literaturecited by 20
  12. 2023Top-Down Visual Attention from Analysis by Synthesiscited by 19
  13. 2022Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problemcited by 18
  14. 2021LPF: A Language-Prior Feedback Objective Function for De-biased Visual Question Answeringcited by 17
  15. 2024Learning from Observer Gaze:Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognitioncited by 17
  16. 2021GraghVQA: Language-Guided Graph Neural Networks for Graph-based Visual Question Answeringcited by 16
  17. 2021VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answeringcited by 16
  18. 2022Attribution-based XAI Methods in Computer Vision: A Reviewcited by 16
  19. 2025Where do Large Vision-Language Models Look at when Answering Questions?cited by 15
  20. 2021A First Look: Towards Explainable TextVQA Models via Visual and Textual Explanationscited by 14
  21. 2021Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answeringcited by 14
  22. 2023VQA Therapy: Exploring Answer Differences by Visually Grounding Answerscited by 14
  23. 2021Connecting What to Say With Where to Look by Modeling Human Attention Tracescited by 13
  24. 2022A Thousand Words Are Worth More Than a Picture: Natural Language-Centric Outside-Knowledge Visual Question Answeringcited by 13
  25. 2022VisFIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectivescited by 12
  26. 2023HaVQA: A Dataset for Visual Question Answering and Multimodal Research in Hausa Languagecited by 12
  27. 2024ViSaRL: Visual Reinforcement Learning Guided by Human Saliencycited by 12
  28. 2021Visual Question Answering: which investigated applications?cited by 11
  29. 2022Joint Answering and Explanation for Visual Commonsense Reasoningcited by 11
  30. 2021Visual explanation of black-box model: Similarity Difference and Uniqueness (SIDU) methodcited by 10
  31. 2021Passive Attention in Artificial Neural Networks Predicts Human Visual Selectivitycited by 10
  32. 2022Prophet Attention: Predicting Attention with Future Attention for Image Captioningcited by 10
  33. 2021Answer Questions with Right Image Regions: A Visual Attention Regularization Approachcited by 9
  34. 2024Uncovering the Full Potential of Visual Grounding Methods in VQAcited by 9
  35. 2024Trends, Applications, and Challenges in Human Attention Modellingcited by 9
  36. 2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
  37. 2022Weakly Supervised Grounding for VQA in Vision-Language Transformerscited by 8
  38. 2023Measuring Faithful and Plausible Visual Grounding in VQAcited by 8

50 most cited of 111 All 111 (JSON)